Camera for locating hidden objects

ABSTRACT

A distance substantially between a camera and an object is measured preferably with a rangefinder. Positional coordinates including an altitude of the camera are determined. A pose including pitch and azimuth of the camera directed at the object is determined from sensors. Positional coordinates of the object are determined using at least the positional coordinates of the camera, the pose of the camera and the distance substantially between the camera and the object which are used to determine a location volume. A database is searched for objects located at least partially inside the location volume. The camera is part of a computing device with a screen. Search results are listed on the screen and an outline of a hidden object in the location volume is drawn on the screen.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. Non-Provisionalapplication Ser. No. 14/147,569, filed Jan. 5, 2014, which is acontinuation-in-part of U.S. Non-Provisional application Ser. No.13/844,626, filed Mar. 15, 2013, now U.S. Pat. No. 9,171,221 issued onOct. 27, 2015, which are both incorporated herein by reference in itsentirety. Application Ser. No. 14/147,569 claims the benefit of U.S.Provisional Application No. 61/819,634, filed May 5, 2013, of U.S.Provisional Application No. 61/823,375, filed May 14, 2013, of U.S.Provisional Application No. 61/825,069, filed May 19, 2013, of U.S.Provisional Application No. 61/864,632, filed Aug. 11, 2013, which areall incorporated herein by reference in their entirety.

BACKGROUND OF THE INVENTION

The present invention relates to digital image devices. Morespecifically, it relates to a camera which is used by a user to find andtrack an object that is hidden from view by another object or by anobstacle.

There is a need to locate an object or person that is hidden from viewor that is difficult to find in a crowd of objects or persons. An objector person that needs to be located may be on a different altitude orheight than a device searching for the object or person. However, thereis a lack of devices, systems and methods that provide locating a hiddenobject. Accordingly, novel and improved systems, devices and methods arerequired that perform the locating of objects that are hidden fromdirect view.

SUMMARY OF THE INVENTION

In accordance with an aspect of the present invention an imaging systemis provided to locate an object, comprising: a first computing deviceincluding a housing, a processor, a camera, a display with apredetermined part to display at least part of the object, a device thatgenerates geospatial coordinates including an altitude, a digitalcompass to generate a pointing direction of the camera, a device whichdetermines an inclination of the camera and a communication circuit; asecond computing device in a housing attached to the object, the secondcomputing device, which is portable and mobile, includes a processor, adevice that generates geospatial coordinates including an altitude, anda communication circuit; wherein the second computing device isconfigured to receive a request originating from the first computingdevice to provide data related to the location of the object, and torespond with data that enables the first computing device to guide thecamera into a pose that places the object in a field of vision of thecamera.

In accordance with a further aspect of the present invention an imagingsystem is provided, wherein the first computing device is mobile andportable.

In accordance with yet a further aspect of the present invention animaging system is provided, wherein the housing of the first computingdevice is a wearable headframe.

In accordance with yet a further aspect of the present invention animaging system is provided, wherein the second computing device isportable and mobile.

In accordance with yet a further aspect of the present invention animaging system is provided, wherein the housing of the second computingdevice is a wearable headframe which includes a camera and a display.

In accordance with yet a further aspect of the present invention animaging system is provided, wherein geospatial spatial coordinatesgenerated by the first and the second computing device are applied by aprocessor to determine the pose that places the object in the field ofvision of the camera.

In accordance with yet a further aspect of the present invention animaging system is provided, wherein the camera is attached to anactuator controlled platform that places the camera in the pose thatplaces the object in the field of vision of the camera.

In accordance with yet a further aspect of the present invention animaging system is provided, wherein the camera of the first computingdevice is enabled to point at a second object, the display of the firstcomputer device is configured to also display images generated by thecamera of the second computing device, the first computer device isconfigured to provide instructions to the second computing device toplace the camera of the second computer device in a pose that generatesan image of the second object on the predetermined part of the displayof the first computing device.

In accordance with yet a further aspect of the present invention animaging system is provided, further comprising the processor of thefirst computing device configured to determine geospatial coordinates ofthe second object.

In accordance with yet a further aspect of the present invention animaging system is provided, further comprising the processor of thefirst computing device configured to determine geospatial coordinates ofthe second object after it has moved to a next position.

In accordance with yet a further aspect of the present invention animaging system is provided, further comprising a remote control with aseparate housing communicatively connected to the first computing devicethat is configured to initiate the instructions to the second computingdevice.

In accordance with another aspect of the present invention an imagingsystem is provided with a first wearable headframe, including aprocessor with memory, a camera, a display configured to display animage generated by the camera, a device that generates geospatialcoordinates including an altitude, a digital compass to generate apointing direction of the camera, a device which determines aninclination of the camera and a communication circuit and a secondwearable headframe, including a processor with memory, a camera, adisplay configured to display an image generated by the camera, a devicethat generates geospatial coordinates including an altitude, a digitalcompass to generate a pointing direction of the camera, a device whichdetermines an inclination of the camera and a communication circuit, thecommunication circuits enabled to exchange data with each other,comprising: the processor with memory in the first wearable headframeconfigured to retrieve instructions from the memory to generate arequest for location data of the second wearable headframe and theprocessor with memory in the second wearable headframe configured toprocess and to authorize the request for location data of the secondwearable headframe and to generate data that enables the camera of thefirst wearable headframe to be placed in a desired pose that places thesecond wearable headframe in a field of vision of the camera of thefirst wearable headframe.

In accordance with yet another aspect of the present invention animaging system is provided, wherein a difference between an actual poseand the desired pose of the camera of the first wearable headframe isdetermined and an instruction is displayed on the display of the firstwearable headframe to minimize the difference.

In accordance with yet another aspect of the present invention animaging system is provided, further comprising the first wearableheadframe is configured to be moved towards an object, the camera of thefirst wearable headframe configured to obtain an image of the object,the display of the first wearable headframe configured to display theimage of the object and the display of the first wearable headframeconfigured to display an image of obtained by the camera of the secondwearable headframe, the second wearable headframe is configured to bemoved towards the object, the camera of the second wearable headframeconfigured to obtain an image that configured to be displayed on thedisplay of the second wearable headframe, the display of the secondwearable headframe configured to display the image of the objectobtained by the camera of the first wearable headframe and the processorof the first wearable headframe is configured to generate a confirmationmessage for the processor of the second wearable headframe when theobject in the image obtained by the camera of the first wearableheadframe and displayed by the display of the first wearable headframeis determined to coincide with the image obtained by the camera of thesecond wearable headframe which contains the object and displayed by thedisplay of the first wearable headframe.

In accordance with yet another aspect of the present invention animaging system is provided, wherein geospatial coordinates including apose of the camera of the first wearable camera are determined,geospatial coordinates including a pose of the camera of the secondwearable camera are determined and geospatial coordinates of the objectare determined.

In accordance with yet another aspect of the present invention animaging system is provided, wherein the camera of the first wearablecamera tracks the object while it is moving and the processor of thefirst wearable headframe provides instructions to the processor of thesecond wearable headframe to track the moving object.

In accordance with yet another aspect of the present invention animaging system is provided, further comprising a platform based camerawith processor controlled configured to place the camera in a desiredpose.

In accordance with yet another aspect of the present invention animaging system is provided, wherein the platform based camera is placedin the desired pose by the camera in the first wearable headframetracking the object.

In accordance with yet another aspect of the present invention animaging system is provided, wherein the object includes a device togenerate geospatial coordinates. In accordance with yet another aspectof the present invention an imaging system is provided, wherein theobject follows a path with known geospatial coordinates.

DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B illustrate in diagram of a camera in accordance with anaspect of the present invention;

FIG. 2 illustrates a system with a network and connected cameras inaccordance with various aspects of the present invention;

FIG. 3 illustrates two tracking cameras in accordance with an aspect ofthe present invention;

FIG. 4 illustrates a display screen of a camera in accordance with anaspect of the present invention;

FIG. 5 illustrates two tracking cameras in accordance with an aspect ofthe present invention;

FIG. 6 illustrates a display screen of a camera in accordance with anaspect of the present invention;

FIG. 7 illustrates tracking cameras in accordance with an aspect of thepresent invention;

FIGS. 8, 9, 10, 11 and 12 illustrate object tracking in accordance withvarious aspects of the present invention;

FIG. 13 illustrates a camera in accordance with an aspect of the presentinvention;

FIGS. 14 and 15 illustrate a tracking scene in accordance with an aspectof the present invention;

FIGS. 16-19 illustrate a platform in accordance with an aspect of thepresent invention;

FIG. 20 illustrates a tracking scene in accordance with an aspect of thepresent invention;

FIG. 21 illustrates a camera screen in accordance with an aspect of thepresent invention;

FIGS. 22 and 23 illustrate a tracking scene in accordance with an aspectof the present invention;

FIG. 24 illustrates camera screens in accordance with an aspect of thepresent invention;

FIGS. 25, 26, 27 and 28 illustrate a tracking scene in accordance withan aspect of the present invention;

FIGS. 29 and 30 illustrate camera screens in accordance with an aspectof the present invention;

FIG. 31 illustrates a tracking scene in accordance with an aspect of thepresent invention;

FIG. 32 illustrates an image frame in accordance with an aspect of thepresent invention;

FIGS. 33-34 illustrate tracking aspects in accordance with variousaspects of the present invention;

FIG. 35 illustrate camera rotation planes in accordance with variousaspects of the present invention;

FIGS. 36-37 illustrate calculation of intersection coordinates inaccordance with various aspects of the present invention;

FIG. 38 illustrates tracking of an object with an aircraft in accordancewith various aspects of the present invention;

FIG. 39 illustrates in diagram a headframe based camera system;

FIGS. 40 and 41 illustrate in diagram a headframe based camera system inaccordance with one or more aspects of the present invention;

FIG. 42 illustrates a camera controller in accordance with one or moreaspects of the present invention;

FIG. 43 illustrates a camera screen in accordance with one or moreaspects of the present invention;

FIG. 44 illustrates a processor based system to perform steps andprocess signals in accordance with various aspects of the presentinvention;

FIG. 45 illustrates in diagram a device to provide geospatialcoordinates and pose coordinates in accordance with various aspects ofthe present invention;

FIG. 46 illustrates a network;

FIG. 47 illustrates a scene reviewed by a headframe camera in accordancewith one or more aspects of the present invention;

FIG. 48 illustrates consecutive screens on a display in accordance withone or more aspects of the present invention;

FIG. 49 illustrates steps performed in accordance with one or moreaspects of the present invention;

FIGS. 50, 51, 52 and 53 illustrate screens on a display in accordancewith one or more aspects of the present invention;

FIGS. 54, 55, 56 and 57 illustrate one or more scenes being reviewedwith a device in accordance with various aspects of the presentinvention;

FIGS. 58, 59A, 59B and 60 illustrate cameras and computing devices inaccordance with one or more aspects of the present invention; and

FIG. 61 illustrates a motorized pan/tilt camera platform in accordancewith one or more aspects of the present invention.

DETAILED DESCRIPTION

In accordance with an aspect of the present invention a wireless,portable and mobile computing device 100 is provided with has a housing112, which may be a single housing. The device 100 has an antenna 116for wireless communication to receive and to transmit signals with theoutside world. The communication circuitry 114 is included to manage andconduct communication with the outside world, for instance wirelesslythrough antenna 116 or via a wired connection through a connector 114.The device has a processor 102 with memory 104. Memory 104 is enabled tohold and retrieve data. Data may be instructions, data upon which theprocessor will operate or data provided by processor or by one or moreof the circuits 108 included in the device, including but not limited toGPS circuitry, accelerometer circuitry, magnetometer, digital compass,and MEMS gyroscope as are currently available on the market as digitallyreadable device which provides data to a memory and are available insystems such as smart phones and mobile tablets.

Further included is a display screen 118. Also included is an inputdevice 110 that can control an object or image on the screen 118 andalso provides signals that can be transmitted to an external device. Akeyboard 108 is also included. FIG. 1A illustrates the aspects of thedevice in one view of the device. FIG. 1B illustrates the device from adifferent view and provides a lens 122 of a camera included in themobile device.

The mobile device is enabled in one embodiment to communicate directly,uniquely with a second device 202 via a wired or wireless channel 208 asillustrated in FIG. 2. In a wired communication channel 202 thecommunication is limited to the two devices 100 and 202 by virtue of adirect cable. In the wireless case a coding or encryption such as in aBluetooth communication provides the unique exchange of data.

In a further embodiment of the present invention a network 200 isprovided wherein the devices 100 and 202 communicate over 200 throughconnections 204 and 206 respectively. In one embodiment of the presentinvention, both 100 and 202 have an object 220 in a field of vision of acamera that is part of the respective devices. However, at a certainconfiguration devices 100 and 202 may be out-of-reach for a directcommunication. In that case the network 200 is used. Network 200 in oneembodiment of the present invention is a cellular phone network. Network200 in one embodiment of the present invention is the Internet andchannels 204 and 202 are for instance Wi-Fi channels that provide thedevices access to the network 200. Other configurations are possible andare fully contemplated. Accordingly, device 100 and device 202 areenabled to perform two-way communication which includes exchange ofsetting date, internal data, instructions, and video and audio data in awireless manner and in real-time or substantially in real-time, wherein“substantially” means within a delay of preferably less than 1 second,more preferably less than 0.1 send and most preferably less than 0.01second, wherein an image is a part of a video image and is a frame ofsuch video-image.

Real-time for images means that the above delays apply to imagesgenerated in real-time on one device and displayed on a screen on theother device. An image is understood to be a frame of a video image.Standard video frame rates are for instance at least 25 or 30 frames persecond; at least 50 or 60 frames per second or 100 to 120 frames persecond. Slower frame rates are possible, for instance as low as at least6 frames per second. Even lower rates (for instance 1 frame per second)are possible though the video image becomes jerky and start to look likea series of static images. However, under severe constraints of noiseand bandwidth the low frame rates as low as 1 frame per second or even 1frame per 5 seconds may be used for object tracking.

In one embodiment of the present invention one or more mobile devicessimilar to 100 or 202 are connected to the network or in directcommunication in real-time or substantially real-time with 100 and/or202 with object 220 in a field of vision.

The object 220 is in one embodiment of the present invention a movingobject or person. The object 220 is in one embodiment of the presentinvention a static object or person, at least initially.

In one embodiment of the present invention a device 210 similar to 100is connected via connection 212 which may be wired or wireless tonetwork 200 to device 202. Device 210 is remote and does not have object220 in its field of vision. While the communication between 202 and 210is also substantially real-time a delay may occur and may be as high as2 seconds or even 4 second or 6 seconds.

In one embodiment of the present invention a server 240 is connected tothe network 200. While in one embodiment cameras 100 and 202 may performthe processing required for tracking as explained above and below, theinstructions required to calculate intersections points, coordinates andimage tracking and detection may be performed on server 240, which isaccessible by the cameras via network 200.

In one embodiment of the present invention, an area is an area undersurveillance by calibrated cameras. Geospatial coordinates of almost anypoint on a surface in the area under surveillance and visible by acamera can be determined from images generated by the cameras in thearea under surveillance. In one embodiment of the present invention auser has a camera with GPS and camera pose determining capabilities andthe user is enabled to view on a screen on the camera an image generatedin real-time or substantially in real-time by one or more calibratedcameras that observe the area under surveillance. Because the 2D spaceof the camera(s) is known, the intersection point of the two cameras(the static camera and the mobile camera) is determined. The 3D spaceunder surveillance is reversibly associated with the calibrated cameras.This means that all surface coordinates in the 3D space undersurveillance are associated with the calibrated cameras that cover thesecoordinates. This means that when the mobile camera tracks an object ora person after a first intersection point has been determined the objectwhich is tracked by the mobile camera can also be viewed on the imagesof the static cameras. For every image frame, geospatial coordinates ofthe object are determined. Each set of geospatial coordinates isassociated with the 3D-to-2D transformation of one or more cameras.

In accordance with an aspect of the present invention the images of themobile and the static cameras are recorded. A time stamp of all thecameras is recorded so that each frame of each camera taken at the sametime has the same time stamp. The geospatial coordinates of theintersection of the pointing direction of the mobile camera with asurface are recorded. The coordinates may be corrected for the size ofan object.

The camera in one embodiment of the present invention is a calibratedcamera and a calibration matrix of the camera has been determined and isstored in the memory. It is well known that a standard camera is aprojective system that projects a 3D point in the real world via anon-ideal 3D:2D projection to a 2D point in the coordinates of the imagesensor in the camera. The non-ideal projection includes a non-idealprojective lens with distortion and rotational effects of the sensorcoordinates relative to the projective plane. The calibration matrixallows the processor to calculate a relationship between a location ofan object in the image plane relative to its actual size and distancefrom the camera.

The literature provides different forms of the calibration matrix. Oneform is:K=T _(im) S _(pix) P _(cam) R _(cam) T _(cam) x.

Herein T_(im), S_(pix) and P_(cam) are the intrinsic camera arrays withT_(im) a translation to the image center, S_(pix) a scaling to pixelsize and P_(cam) the perspective projection that is performed by thecamera. R_(cam) and T_(cam) are extrinsic parameters with R_(cam) arotation array of the camera center relative to the camera and T_(cam) atranslation or camera location relative to the object. The symbol xrepresents the real-world 3D coordinates. The above notation is usedfrom a presentation of University of Jyväskylä at <URL:http://users.jyu.fi/˜tro/TIES411_08/camera.pdf> which is incorporatedherein by reference. How to estimate the calibration components isdescribed for instance in an on-line article at <URLhttp://www.umiacs.umd.edu/˜ramani/cmsc828d/lecture9.pdf> by Dr. RamaniDuraiswami of the University of Maryland, which is also incorporated byreference herein.

The screen 118 shows an image captured by the camera and acts in oneembodiment of the present invention as a viewer of the camera. A window120 may also be displayed on the screen 118 and for instance highlightsor identifies an object captured by the center part of the camera.

Embodiments of the present invention in one aspect are intended to allowone camera in a first location operated by a user to focus on an objector a person, to have a second camera either operated by a user or fixedon a movable platform to also focus on the object; to have the firstcamera track the object or person move in space, preferably in aconstant plane or in a moving plane either by rotation or translation ofthe plane or both and based on images and on other data at leastfacilitating the second camera to track the object or person based ondata provided by the first and second camera.

At Least Two Cameras are User Operated

In this embodiment of the present invention a first camera is pointed bya user at an object or at a location, the object having GPS coordinatesand a size that are unknown, but that can be estimated. The user of asecond camera obtains the object in view of the second camera.

This is illustrated in FIG. 3. Cameras 202 and 204 are pointed at anobject 220 302 and 304 are the imaging planes of the cameras. Camera 202has GPS coordinates GPS2 (which may include an altitude or elevation ofthe camera) generated by the GPS unit in the camera; an height h2, whichmay be measured, estimated or provided by the GPS unit; an azimuth az2generated by the digital compass; and an inclination incl2 relative tothe horizon determined by the accelerometers or gyroscopes orinclinometer. Camera 204 also has known coordinates.

The cameras have a screen 400 to view the object as illustrated in FIG.4. The screen has a window 406 which in one embodiment of the presentinvention is smaller that the size of the screen. Accordingly, thewindow 406 captures only a part of the total image generated by thecamera of a scene.

The user will move the camera in such a manner that the image of theobject appears inside the window. For camera 202 the image of object 220is 402, for camera 204 the image of object 220 is 404. Each cameraexperiences a different perspective distortion. Through a communicationchannel the cameras will also receive and display the image recorded bythe other camera. Accordingly, camera 204 displays images from camera202 and from itself and camera 202 displays images from itself and fromcamera 204. To distinguish between own images and other images the otherimages may be displayed in a transparent mode overlaid on an opaque modeof images generated by the camera to which the display corresponds. Inone embodiment of the present invention the entire image taken by thesecond camera, which may be called a remote camera, is displayed on thescreen of the first camera. In one embodiment of the present inventionthe remote image or part thereof is transparent and the own image orpart thereof is opaque. In one embodiment of the present invention theown image or part thereof is transparent and the remote image is opaqueon the screen of the first camera.

In one embodiment of the present invention the role of the first and thesecond camera are reversed and the second camera becomes the leadingcamera. In one embodiment of the present invention the screen of thefirst camera can be operated in a remote mode so that the imagegenerated by the second camera is displayed on the screen correspondingor of the first camera. All roles of the computing devices with camerascan be changed and assumed by other cameras.

The terms first and second camera (or even third and remote camera) willbe used herein. It is to be understood that the term camera herein isintended to mean a camera with one or more lens and at least one imagesensor being part of a mobile computing device such as a smart phonewith GPS and other facilities as described above, unless indicatedotherwise.

Several scenarios are possible and will be dealt with.

Scenario 1: the first and second cameras are each operated by its ownuser and have the target object 220 within a focus window.

This situation is illustrated in FIG. 5. For both cameras GPScoordinates are available including an elevation, an azimuth and aninclination which are represented by GPS2, h2, az2, incl2 for camera 202and GPS4, h4, az4 and incl4 for camera 204. The coordinates GPSO forobject 220 can be easily calculated from by the processor in the camera(=mobile computing device) using simple triangulation using thecoordinates of the cameras. It is assumed that both cameras 202 and 204have an image of object 220 inside the window 406 on the screen of thecamera.

It may turn out that the object moves with a certain speed to a position520 as illustrated in FIG. 5. FIG. 6 illustrates a screen on a mobiledevice with a camera (either 202 or 204). No camera is in the lead. Thescreen shows the window 606 on a screen 600 showing the entire imageswith the image parts of the object inside the window, as the users eachpoint the camera at the object. The own camera has for instance theopaque image and shows the image of the other camera in transparentform. The screen provides further data. At the left side is the data ofcamera 202, including an ID as item 601, data window 603 showing forinstance the coordinates of camera 202 and a gauge type display 605showing a predicted movement of camera 202. One can see from FIG. 5 thatobject 220 is moving almost straight away from camera 202, In that casecamera 202 should be held by the user in an unchanged position.

In one embodiment of the present invention a screen on a camera displaysmanagement and status information as illustrated in FIG. 6. The window606 shows the images recorded by cameras 202 and 204 in certain stagesof transparency. A user may actually set the level of transparency foreach of the images, for instance based on the fact if the camera isleading or following.

A user may select or remove icons and data from the screen by a controlbutton 619, which may be a part of the screen if that part is touchsensitive.

On the left side information related to camera 202 is shown alsoincluded an inclination gauge 615 which indicates that the camera has tobe tilted upward to continue to track object 220 at the estimated speedand direction. A similar set of indicators is provided for camera 204with ID 607 a predictive turn indicator 609, a window with cameracoordinates 608 and an inclination indicator 617, which indicates thatthe camera has to be tilted down at the estimated speed and direction ofobject 220. Indicator 611 indicates a slight change in absolute courseof object 220. Window 610 shows the estimated new coordinates of theobject.

In one embodiment of the present invention at least window 606 displaysa map of the area under surveillance by the cameras. A mark shows theposition of object 220 on the map. An arrow or another mark shows wherethe object is expected within a certain time period.

If the movement of the object on the screen is relatively slow, it iseasy to cycle between different display screens while still maintainingan image of the object inside window 606. This allows a user to view themanagement and indicator aspects of the system on the screen in asequential manner.

In one embodiment of the present invention object 220 has moved to theright and is in position 720 as illustrated in FIG. 7. At a certainstage object 220 moves out of range of camera 202 and a new camera 702will start tracking what is now called object 720. In one embodiment ofthe present invention a hand-off method is provided. For instance whencameras 202 and 204 have a compass angle that reaches a threshold analert is generated for a hand-off.

As part of a project cameras are assigned a number of engagement forinstance along a route or path that the object is expected to move. Eachcamera is provided with a circle of operation. Each camera has its owncoordinates (GPS, azimuth, elevation, pose or inclination) that areknown. The circle has the camera as its center and an operational rangeas its radius.

Based on overlap of the circles and a projected direction or trajectoryof the object cameras are dynamically assigned by a computer to one ofindividual cameras that are part of a system. This is illustrated inFIG. 8. An object 820 can follow one of at least 2 trajectories 806 and808. The area contains a plurality of cameras 801, 802, 803 and 804which are spread over the area and which are (as stated before and stillmaintained) cameras in mobile computing devices with potentially all ofthe features as required earlier. One can see that cameras 801, 802 and803 can cover trajectory 808 and cameras 801, 802 and 804 can covertrajectory 806. In one embodiment of the present invention at least onecamera in a trajectory gets a management position. This is for instancethe last or second to last active camera at a certain time that recordsthe object. The cameras are all “signed-in” to the system and areidentified by an identifier as well as their coordinates (such as GPScoordinates). The active camera, which also knows its own location,determines the areas that the object is estimated to traverse andidentifies the next camera that is most likely to cover the movingobject 820.

For instance camera 801 is active and tracks (or the user of 801 tracks)the object 820. Camera 802 in this scenario was already engaged to bethe next active camera. Based on the estimated trajectory, assume it istrajectory 806, the processor in 801 calculates that camera 804 is themost likely able to track 820 along 806 and a message is sent to 804 toalert that it will become one of the active cameras. It will receive allrelevant data about the trajectory and speed. And it will receive apointing direction and an inclination where to point the camera tocorrectly point at the oncoming object 820. A time alert, based on thecalculated speed of the object may be provided or calculated, indicatingthe user of 804 to point the camera at the right point at the right timea start recording the object. Furthermore, the alerted camera receivesupdated information where to point the camera, for instance using thescreen of FIG. 6. Camera 803 may be alerted that its services are notrequired.

Accordingly, the cameras (or the devices that contain the cameras)become replicated, replicating and updated intelligent participants in achain of cameras. In one embodiment of the present invention the camerasare opted in to such a system. They contain all the required functionsto participate and they have a channel with other participating camerasin the chain. In one embodiment of the present invention an activecamera broadcasts a request to all opted in cameras to the system for aresponse if the contacted cameras are in a desired area along anestimated trajectory. Only the cameras that fulfill the requirementsrespond with their identifiers and coordinates. The requesting cameraactivates a program that determines which camera is a best fit (forinstance the closest camera, or the camera that is on the most likelytrajectory) and starts a communication to inform the selected camerathat it is selected and transmits all required data to that camera.

In one embodiment of the present invention the communication between thecameras is automatic. Because in this scenario cameras are useroperated, a user confirmation is required from the selected camera.

In one embodiment of the present invention a server based system isprovided which is illustrated in FIG. 9. A plurality of camerasincluding 902, 904, 906 and 908 which are cameras in mobile computingdevices as described earlier is connected to a network 900. Preferablythe connection is a wireless connection, but a wired connection is alsopossible. The network 900 includes the Internet. Also connected tonetwork 900 is a server 910 which runs at least a sign-up program for acamera tracking system and may have the request, alert and assigningcapabilities that are available in distributed form in the systemillustrated in FIG. 8.

The system of FIG. 9 is applied in different situations including whenthe route 901 that an object 920 will follow is known. Such situationsare known, for instance in situations of running a marathon or a parade.The server 910 hosts the required software and is configured to sharesoftware for the cameras with the cameras or with other devices. Theserver 910 may be part of a social network with members or a specifictracking service. Connected to a network is also a computer 912.

A person may sign-up individually with a camera or with a plurality ofcameras such as 902, 904, 906 and 908 to be part of a tracking system.The sign-up takes place for instance via computer 912 or via one of thecameras, which is in essence a computing device. Computer 912 mayrequest a tracking service from 910, for instance related to a specificevent. In response to such a request for a specific race on a specificdate a unique identifier is generated and assigned to that requester.The requester submits a request for “tracking cameras” related to aracer on the social network web site. Friends of that person can see therequest and can sign-up to participate, preferably via their mobiledevice with the camera. Each signed up camera receives the propersoftware from the server and a unique identifier associated with therequester's unique identifier.

During the event, the first camera in a series of cameras startsrecording object 920 and alerts or activates the next cameras in theseries of cameras. Video images generated by the cameras of object (orperson) 920 are transmitted to servers 910 and are available, preferablyin a combined form, on the social network website.

Cameras such as 902, 904, 906 and 908 are pre-installed along a route901. The cameras have a field-of-view that keeps a moving object in viewfor at least 5 seconds, 10 seconds or 30 seconds or longer. The camerasare calibrated so that the 3D space of the route, or at least the partwhere the object or person is moving can be correctly identified on the2D screen. A computing device that has the calibration data and receivesthe 3D location of an object in the field-of-view of the calibratedcamera, thus is enabled to draw a box in screen space identifying anarea related to the 3D coordinates.

The object 920 has a transponder 922 with a unique identification whichmay be a GPS transponder or a positional RFID device. In one embodimentof the present inventions a series of receivers 930 along the routereceive real-time and accurate location information about the positionof the object or person carrying the GPS device.

Each camera along route 801 has associated with it its locationcoordinates and the coordinates of the area its field-of-view it covers.Each image of a camera is associated with a timing moment and withcoordinates it covers. Concurrently, server 910 records the GPS data andtime of the GPS device or location transponder 922.

A user retrieves, for instance on computer device 912, images of therace that were stored. By applying the unique identification of thetransponder, the server retrieves only images that contain 920 (and thus922) in the field-of-view of a camera. This is illustrated in FIG. 10 inan image 1004 generated by camera 904 at a defined race at a definedlocation at a defined time. Person 920 is carrying location device 922.A person 1002 is also in the field of view of the camera. By definingthe ID of 922 and a time, the proper images of camera 904 are identifiedas having 920 in it. Alternatively, by defining an ID and a locationserver 910 also retrieves the correct images.

The server 910 identifies the runner 920 by selectively drawing a box1101 around the correct person based on stored GPS data as isillustrated in FIG. 11. Device 922 may be a phone such as a cellularphone and a location is determined by triangulation of the device fromsignals that device 922 emits.

Participants in a ceremony are carrying a transponder that identifiestheir location in a field of view of a calibrated camera. The camera maybe a wide-field camera generating an image as illustrated in 1200 ofFIG. 12. The scene may also be covered by a plurality of cameras togenerate 2 or more images. For instance an image 1203 while a personapproaches the center of a stage, image 202 when at the center of astage and image 1201 while leaving the stage. In a similar way asdescribed above, the images are stored on a server, for instance of asocial network and are retrieved by applying person identification.

At Least One User Operated Camera Needs Guidance

One camera has an image of an object within a window and has to guideanother user with a camera to obtain the same object in the window. Thisis illustrated in FIGS. 3-5. A camera 204 has an object 220 in a window406 on a screen of the camera. A user of camera 202 points the camera ingenerally the right direction, but is unable to identify the correctobject. The user of camera 204 guides the user of camera 202 to theobject. This is illustrated in FIG. 13. The screen 1300 of camera 204has image 1304 of the object in window 1306. It shows that image 1302 ofthe object taken by camera 202 is outside and to the left of the window1306. This means that the pose of camera 202 has to be adjusted bymoving camera 202 into the direction of the object. Camera 204 isprovided with a control 1312 which may also be used to control a cursoron the screen of the device 204. The user of camera 204 pushes on theleft arrow of control 1312 which will highlight indicator 1308 (whichmay be provided with a camera indicator) indicating to the user ofcamera 202 to rotate camera 202 to the left. An indicator 1310 indicatesif the camera 202 has to be rotated to the right. Also up and downrotation indicators may be provided. In accordance with an embodiment ofthe present invention, guidance may take place with a separate guidancedevice 4200 in remote contact with the camera and with the remotecamera. This is illustrated in FIG. 42 and will be described later.

By pressing on the center of 1312 or on any other pre-assigned controlsuch as 4200 an “object in focus” indicator on the screen of camera 202is activated indicating to the user of camera 202 by the user of camera204 that the camera 202 is placed in the correct and desired pose. Theuser of camera 202 also confirms the correct pose and the coordinates ofthe crossing of the two viewing directions of cameras 202 and 204 iscalculated based on GPS and other coordinates such as azimuth andelevation.

In one embodiment of the present invention determined cross viewinglocations are stored with their respective pose (including coordinates)on a memory in at least one of the cameras or on a server that isaccessible by the camera.

For instance a user of a camera can walk around an environment andidentify and store coordinates of locations on the camera. This allowsthe user of the camera to recall the location from the memory, determinea current location of the camera and have the processor based on therecalled location and current location calculate the required pose ofthe camera to be pointed at the recalled location from the currentlocation.

In one embodiment of the present invention a map 1408 of the environmentis stored on the camera and can be made visible on the screen. This isillustrated in FIG. 14. An object 1414 may be spotted on a crossing of astreet. Because a map is calibrated against GPS coordinates thecoordinates of 1414 are known and can be calculated by a camera 1404which can share them with 1402.

In a further embodiment of the present invention known objects 1410 and1412 with known coordinates are within a neighborhood of object 1414.For instance one can via an input mechanism inform the processor of 1404that object 1412 is closer to the camera than 1414 and object 1410 isfarther away. Camera 1404 determines the azimuth of 1414 and an estimate(for instance a circle) of a distance to the camera 1404.

This is further illustrated in FIG. 15. The estimated distance isillustrated by the circle 1501. This translates for the camera 1402 into(in this case) an uncertainty in azimuth which is indicated on thescreen of 1502 by a widening of the window 1504. One can be reasonablycertain that the object 1414 based on the estimated distance to camera1404 is located within the window 1504 of camera 1402 when the camera1402 has an azimuth pointed at the center of the circle 1501. Suchestimates are helpful when camera 1402 or a user of 1402 has troublelocating or identifying 1414 and has to rely on camera 1404. When object1414 starts to move or gets close to a known object such as 1410, moreprecise coordinates can be determined.

In a further embodiment of the present invention an estimate can be madeof the size or height of the object. The camera 1404 is a calibratedcamera. Accordingly, a size of an object inside a window (for instancein number of pixels) can be immediately translated to a distance. Incase of a linear calibration a height of 40 pixels may correspond to asize of 2 meter. A change in image size to 20 pixels immediate indicatesa change in distance of the object to the camera with a factor 2. It isknown that lens calibration may not be linear. A processor is programmedto adjust for non-linear lens distortion.

In one embodiment of the present invention a location with GPScoordinates including an altitude for an object are determined and areprovided to two cameras. The screens of the cameras as illustrated inFIG. 6 show that a camera can be pointed in the right direction when thecorrect azimuth and inclination or pose are provided to the camera, evenif a user of the camera does not know at what object he is looking. Itmay be that the user of the other camera knows what he is looking at. Inone embodiment of the present invention the control of one camera may beprovided to the user of the other camera. As long as the user of thefirst camera points the first camera in the correct pose, the user ofthe second camera may apply other first camera controls for instance viathe second camera. This allows the user of the second camera forinstance to apply zooming in or out to the object with optical ordigital zoom on the first camera.

Scenario with One Camera being User Held and Operated and a SecondCamera on an Actuated Platform

In one embodiment of the present invention a computer controlledplatform 1601 enabled to hold the mobile computing device with camera1602 containing at least one lens/sensor unit, the computing devicehaving at least a wired or wireless communication device to communicatewith another computer device with camera 1700 illustrated in FIG. 17with a screen 1701. In case of a wireless communication connection thecomputing device 1602 has an antenna 1603 and the device 1700 has anantenna 1703. The structure 1605 supports the rotatable or movableplatform 1601. The platform 1601 has at least one remotely controlledactuator such as an electrical motor, which may be a stepping motor orany other remotely controllable motor or actuator, which enables theplatform 1601 to be rotated in a remotely controlled fashion by device1700. The actuator is either bi-directional based on a control signalallowing it to move in a direction and a reverse direction, or anactuator contains at least two actuators, one working in a direction andthe at least other one working in the reverse direction.

In one embodiment of the present invention the platform itself has areceiver to receive a control signal for an actuator in the platform,for instance from the computing device or from the display. In yet afurther embodiment of the present invention a rotating control signal isprovided via computing device or camera 1602 to a motor or actuatorcontroller in the platform. In one embodiment of the present inventionboth the platform and the computing device as well as the display areBluetooth enabled.

In one embodiment of the present invention the platform has integratedon it: GPS, inclinometer (such as gyroscope or accelerometer) andcompass capabilities. In one embodiment of the present invention thedevice with camera 1602 has those capabilities. In one embodiment of thepresent invention the camera 1700 has those capabilities. In a furtherembodiment of the present invention the platform 1601 has capabilitiesto communicate with device with camera 1700. In a further embodiment ofthe present invention the computing device with camera communicates withthe platform, for instance for transmitting images in real-time from thecamera to the platform.

In one embodiment of the present invention the platform, for instancedevice holder 1802 has a connector 1808 to make a connection withcomputing device with camera 1801 as illustrated in FIG. 18, wherein anactuator controller in the platform receives a control signal via 1801.Thus, by applying a remote controller a device holder attached to aplatform which in one embodiment contains a computing device with alens/sensor unit is instructed from another a computer device to followan object. In one embodiment of the present invention a computing deviceon a platform rotates substantially in one plane relative to a structurethe platform is attached to. In a further embodiment of the presentinvention a computing device on a remotely controlled platform is alsoenabled to rotate in a second plane, for instance by using a verticalactuator such as a telescoping actuator 1606, or any other actuator thatcan change the orientation of the platform.

The use of a specific connector such as 1808, in one embodiment of thepresent invention requires that a device and a device holder on theplatform having matching or corresponding features, and allows to createa platform or at least a device holder of the platform that isproprietary to a certain device.

In one embodiment of the present invention a remote controller 1702,which may be a joystick, is incorporated in the device 1700. In afurther embodiment of the present invention one or more sensors such asaccelerometers 1704 are attached to the display or are incorporated inthe device 1700. Such a display with accelerometers is disclosed in U.S.Pat. No. 7,688,306 to Wehrenberg et al. issued on Mar. 30, 2010 which isincorporated herein by reference. The purpose of the sensors is todetermine a position or a change of position of the device 1700. Basedon the detected change in position a signal which is associated with thedifference of position of the device 1700 is generated and is used tocontrol or drive a motor or an actuator in the platform. Positionalsensors and positional difference sensors are known and includeaccelerometers, magnetic sensors, MEMS with mechanical components,gyroscopes such as optical fiber gyroscopes, Hall sensors, inertialsensors, vibrational sensors and any other sensor that can be applied todetermine a positional or angular difference or angular rate change orangular velocity or a linear rate change or linear velocity change ofthe display. These sensors will be called a positional sensor herein.Many of these positional sensors are available in very small formats,for instance integrated on one or more chips, which can easily be fittedinside a display, even in an ut/ul display. A change in position orangle of the display causes the actuator in the platform to beactivated.

Gyroscopes and accelerometers in MEMS circuits are marketed by AnalogDevices, Inc. of Norwood Mass. Complete attitude circuits are marketedby Navtek Technologies of Richardson, Tex. Digital compass circuits aremarketed by STMicroelectronics. Complete GPS chips that communicate withrelated pose devices are manufactured by Sony, for instance in the D5600and D5601 chips.

In one embodiment of the present invention, the actuator in the platformis activated based on the amount of change in the display from a neutralposition. To stop the actuator from being activated the display has tobe returned to its neutral position. A larger change in position by thedevice 1700 from its neutral position can create a faster activity bythe actuator. In a further embodiment of the present invention, activityof the actuator depends on the rate of change of the position of thedevice. Also because the sensors are directionally sensitive and theactuator is, in a further embodiment of the present invention, a controlelement such as a button is provided on the device, which allows theactuator to respond or stop responding to a change of position of thedevice. This allows a user to return the device to a neutral positionwithout activating the actuator.

In a further embodiment of the present invention the device with camera1700 has a control element, such as a button, to reverse the movingdirection in a horizontal plane or a substantially horizontal planebased on a movement of the device. Substantially horizontal is within arange of preferably not more than 45 degrees, more preferably not morethan 30 degrees, most preferably less than 15 degrees from the horizon.When the device is moved from left to right the platform also moves fromleft to right. This is useful when a user of the device 1700 and thecamera 1801 in the device holder 1802 face in the same direction. Byactivating a control on the device 1700 a user can reverse the directionof the platform, so that when the device 1700 moves from left to rightthe camera, seen from the direction a lens is facing, moves from rightto left. This is useful when the camera and the user are facing eachother. Activating the control again reverses the direction of theactuator as a result of a movement of the device 1700.

A user confirms in a certain position a zero position which correspondswith a zero position of a computing device. By moving the device 1700with the positional sensors, the user can cause the movement of theplatform and thus of the computing device with a camera and track anobject or pan an area. In one embodiment of the present invention afixed tracking plane for the platform can be set or programmed. In afurther embodiment of the present invention the platform can be providedwith a third movement, for instance in a vertical plane.

Accordingly, a platform 1800 is provided which can permanently orremovably hold with a holder 1802 a computing device with camera 1801.The holder 1802 is movably connected to an arm or second part 1803 ofthe platform via one or more actuators 1806 which can rotate and/or move1802 relative to 1803. A housing 1804 is part of or attached to arm 1803and holds all relevant circuitry. The camera 1801 is communicativelyconnected to circuitry in 1804 via 1808 which may be wireless or wiredcommunication circuitry. Circuitry in 1804 controls actuators 1806.Controlling signals may be provided to 1804 via 18008 via the device1801 which receives data from an external source. In a furtherembodiment of the present invention housing 1804 contains all circuitryto transmit and receive data and may contain a processor and memory andGPS and pose determining circuitry. The platform 1800 can be permanentlyor removably attached to a structure 1805. The structure 1805 may be astatic structure such as a building, a movable structure such as a standor tripod, or a moving structure such as a vehicle, a car, a boat or anaircraft such as a drone.

The actuators may be a controlled mechanism that provides rotation inthree planes for azimuth, pitch and roll. In general two planes (azimuthand pitch) may be sufficient. Rotation in one embodiment is provided bystepper motors which are preferably bidirectional stepper motors. Theplatform has at least one neutral position which can be achieved bygiving a “go to neutral” command. Preferably the platform can be rotatedover a range of plus and minus 60 degrees azimuth and plus and minus 30degrees in pitch from the neutral positions and more preferably fromplus and minus 90 degrees azimuth and plus and minus 60 degrees pitch.In one embodiment of the present invention additional stepper motors areprovided to correct the roll of the platform to zero horizon form adeviation which preferably is not greater than 30 degrees from neutralto the horizon but may be greater. In one embodiment of the presentinvention the roll can be corrected manually by allowing the platform tobe manually rotated about an axis with sufficient resistance so that theplatform does not roll spontaneously from its neutral position.

When the platform is attached to a moving structure such as a vehicle ora drone, the platform based camera is provided with an orientation orpose in such a way that the sum of the vector in real space determiningthe position of the drone and the vector directing the platform basedcamera results in a vector that points to the object that is beingtracked.

In one embodiment of the present invention a user of mobile computingdevice with a camera 2004 tracks an object 2020, which may be a movingobject as illustrated in FIG. 20. A movable platform with a camera 2002is configured to also track the object 2020 after, if required, aninitialization process. Both 2002 and 2004 have the facilities todetermine a GPS location of the respective devices, an azimuth of thelens orientation and an inclination as was already described earlierabove. Camera 2004 has a screen as described earlier and for instanceillustrated in FIG. 6.

Several scenarios are possible. In a first scenario a GPS location‘GPSob’ of the object, including an altitude ‘hob’, is known to device2004. Camera 2004 is focused on object 2004. In accordance with oneaspect of the present invention the operation of the camera, includingfocus and shutter settings, are automatically controlled by theprocessor once the distance to the object has been determined orestimated. The camera 2002 is informed of the ‘GPSob’ of the object andthe camera 2002 is turned in the correct direction of 2020 by theactuator(s) 2005. The correct GPS coordinates including altitude of 2020may be derived from a map, or from two cameras like 2004 or from anestimate by 2004 or by manually rotating 2002 towards 2020. When cameras2002 and 2004 with known orientation are focused on 2020, thecoordinates of 2020 can be calculated.

In another scenario camera 2002 is directed to the object 2020 by thecamera 2004 or the user thereof. In that case the camera 2002 is turnedin the right direction by controls on camera 2004 which control theactuators 2005. It has already been explained that a user of 2004 caninstruct a user of another camera where to direct by using indicators onthe screen. In one embodiment of the present invention the indicatorsthat were intended for the user of the other camera are applied to theactuators 2005, thus allowing a user of 2004 to point the camera toobject 2020. The user of camera 2004 can see the image taken by 2002 onthe screen of camera 2004 and thus knows when the object 2020 iscaptured by 2002 when the image taken by 2002 appears inside a window onthe screen of 2004 and for instance coincides or overlaps with an imageof 2020 taken by 2004 inside the window. The moment the images of 2020taken by 2002 and 2004 overlap inside the window the cameras are bothcorrectly directed and ‘GPSob’ and ‘hob’ can be calculated.

Once the GPS coordinates (including altitude) of the object isdetermined the pose (azimuth and inclination) of the cameras is fixed totrack the object. There are several ways to track the object. In oneembodiment of the present invention the user operated camera is leadingand the platform bound camera follows. There are several ways to achievethis. At least two embodiments will be provided. It is noted that handheld cameras are of course not always held in a stable position, unlessthey are fixed on a structure like a tripod. Hand held cameras are proneto show jitter or shake from the user. Image stabilization techniquesfor creating stable video images are known. Accordingly, allsynchronization of the two cameras will take place on stabilized orstable images from which shake and jitter have been removed. A low passfilter can be used to get rid of low frequency user caused noise incalculated control signals that are provided to the platform basedcamera.

In a further embodiment of the present invention control signals for theactuators are determined based on predictive filtering techniques likeKalman filtering or particle filter techniques. For instance U.S. Pat.No. 8,222,207 to Kenji Baba issued on Jul. 17, 2012 which isincorporated herein by reference teaches particle filter based tracking.US Patent Application Publication 2009/0213219 to Eggert et al.published on Aug. 27, 2009 which is incorporated herein by referencediscloses object tracking using a Kalman filter.

FIG. 21 illustrates one embodiment in accordance with an aspect of thepresent invention. The processor of the hand held camera determines aheight 2104 of the object. This may be done from an extracted feature2102 of the image of the object. However, the height may be adjustedmanually by the user if required. The feature 2102 may be an edge orcorners or any other feature extracted from the image of the objecttaken by the hand held camera. The change in height 2104 of 2102 inconsecutive images indicates if the distance between the object and thecamera is increasing, diminishing or remains constant and an estimatecan be made of the distance and the rate of change of the distance basedon the calibration of the camera.

The distance determined by the camera, combined with the GPS and azimuthof the hand-held camera allows to provide an estimate of the newposition in GPS coordinates of the object. These estimated GPScoordinates of the object are used to calculate an azimuth and aninclination of the platform based camera and are translated intoinstructions for the actuator to direct the camera into the newlycalculated direction. A feature 2106 of the object corresponding withfeature 2102 is extracted from the image of the object recorded by theplatform based camera. The hand held camera and the platform basedcamera are synchronized when 2106 and 2102 are aligned or have somepredetermined distance. The actuators of the platform are instructed tominimize the distance between 2106 and 2102. A height 2108 of the object2106 as seen by the platform based camera is applied to determine achange of distance between the platform based camera and the object.

FIG. 22 illustrates situations wherein the size of an image of theobject 2220 is applied. In one embodiment of the present invention, aninitial position of object 2220 is determined, including distances ofthe object to the cameras 2202 and 2204. A size of the object, at leastin pixels, is determined by each camera. The object moves to position2221. The change in pixel size of the object reflects a change indistance to the camera. The change is distance and new distances arecalculated for each camera. The processors on the cameras determine that2221 is at a distance determined by circle 2209 from 2202 and a distance2207 from 2204. Position 2221 is at the crossing of the circles 2209 and2207 which can be calculated with simple analytic geometry.

Object 2220 moves in a straight line away from platform based camera2202 but moves at an angular speed Δθ for camera 2204. That means thatextracted feature 2106 remains in place but grows smaller whileextracted feature 2102 moves to the right inside the window. Because2106 stays in place in the 2D window space the platform of 2202 does nothave to rotate to keep 2106 in place. However, camera 2204 has to turnat an angular speed of Δθ to keep 2102 in place inside the window space.In general a user will lag in speed in following an object. Theprocessor in one embodiment of the present invention creates a mark 2109to assist a user to bring the camera 2204 in alignment with the movingobject 2220 in such a way that the features 2106 and 2102 are properlyaligned when 2220 has reached position 2221. It is noted that feature2106 follows feature 2102 or camera 2202 follows camera 2204.

In one embodiment of the present invention camera 2202 tracks object2220 autonomously for a very brief period after being initialized to thecorrect positioned by the hand held camera. It does so by tracking thefeature 2106 in its own 2D space and activating the actuators so thatthe feature 2106 maintains its position in 2D space of the screen ofcamera 2202. At intervals, for instance at regular intervals of 10 ms,50 ms, 100 ms or 1 sec, difference between features 2106 and 2102 in 2Dscreen space is determined. In a preferred embodiment of the presentinvention the features 2106 and 2102 overlap or coincide in 2D screenspace when both cameras have the object properly within the screenwindow.

Accordingly, a difference in location of the features in calibrated 2Dscreen space indicates that the camera 2202 is not in synchronizationwith camera 2204. The platform camera in that case can be ahead of theobject and thus has to at least slow down rotation or even reversedirection. The platform camera also can lag the hand held camera and hasto catch up (or increase its rotation speed) if the hand held cameramaintains its speed. Or the platform camera has to maintain its speedfor a calculated time if the had held camera is slowing down until theplatform camera has caught up. A Kalman filter can be applied toestimate a required rotation speed of the platform camera. Operationallyone can create a feedback system such as a PID system that minimizes anerror angle between the hand held camera and the platform camera,wherein a feature in 2D screen generated by the hand held camera isconsidered the actual position.

Accordingly, the processor will match the position of feature of 2106from an image of platform camera 2202 in 2D screen space with feature2102 of hand-held camera 2204 in 2D screen space. This is illustrated inFIG. 23 which illustrates that camera 2202 is rotated to the right.Assuming some delay by the actuators of the platform it may be thatfeature 2106 appears temporarily to the right of 2102 and the actuatorshave to move the camera to move feature 2106 to the left to align with2102 again. A difference between an actual azimuth and a desired azimuthof the platform based camera is determined and used to activate theactuators of the platform. The speed of the actuators, which may bepulse controlled stepping motors depends on the size of the differenceangle Δθ. If the angle is large, for instance 5 degrees or 10 degrees orlarger a faster sequence of pulses will be generated by the controller.If the angle is mall, smaller than 5 degrees or close to 1 degree pulsesfor the stepping motor that rotates the platform in a certain plane willbe generated at a lower rate to prevent overshoot. Furthermore, a lowpass filter may be applied to suppress jitter created by the userholding the hand held camera.

This is further illustrated in FIG. 24. It is noted that both camerasoperate in their own 2D screen space. The screen spaces in oneembodiment of the present invention are normalized so that a user cansee on the screen of hand held camera if and how the platform camera istracking. The user keeps the object in a fixed or substantially fixedposition (for instance within 1 mm, 5 mm or 10 mm variance) on thescreen. This creates a mark on the screen or at least for the processorin 2D screen space. The 2D screen space also has image of the object ora feature of the object as generated by the platform camera. Theprocessor can easily determine if the platform camera is in sync (screen2401) is lagging (screen 2402) and the platform has to rotate to theright or the platform camera is leading (screen 2403) in which case theplatform has to rotate to the left.

The use of a size of an object for tracking is further illustrated inFIG. 25. A camera with camera node has an image plane that is normal tothe azimuth direction towards an object with a size Sr and GPScoordinates (x_(ob),y_(ob),z_(ob)). The camera is located at(x_(c),y_(c),z_(c)). From that one can make an accurate determination ofdistance d_(ob) from the camera to the object. This can take place, forinstance in an initial phase of tracking wherein both the hand held andplatform cameras have known coordinates and are appropriately directedat the object so that for instance the screen of the hand held camerasshows overlapping matching images inside the window on the screen. Evenin a simple approximation using the pin-hole camera equation

$\frac{S_{i}}{f} = \frac{S_{r}}{d_{ob}}$with S_(i) the determined size of the object image in the image plane ofthe camera, S_(r) the to be calculated size of the object, d_(ob) thepreviously determined distance of the camera to the object and f thefocal length of the camera, one can make a good estimate of the realsize of the object.

One can also apply a more detailed transformation of real-space tocamera space based on the camera calibration to adjust for lensdistortion. Accordingly, one can with a preferred accuracy determine theactual size of the object with preferably an accuracy of 10% better ormore preferably at least a 5% accuracy. In most cases the accuracy ofthe object determination will depend on the accuracy of the GPS locationunits, which in standard equipment is about 10 meter or better undergood conditions. A horizontal accuracy of 2-3 meters can be obtainedwith GPS SPS receivers. Augmented GPS such as High Accuracy NDGPS(Nationwide Differential GPS) allows for an accuracy in the 10-15centimeter range.

Broadcom® of Irvine, Calif. has announced a new Location Architecturewith BCM4752 with Multi-Constellation acquisition with sub-meteraccuracy and fast acquisition.

Novel, fast positioning systems allow for the tracking and calculationsrequired herein.

Accordingly, a change in image size can be translated into a change ofdistance of the object relative to the hand held camera and to theplatform based camera. This is further illustrated in FIG. 25. Theobject with length Sr moves to a new position, to the right of itsprevious position and away from the camera. The user will turn thecamera so that the new image of the object with diminished image sizeSinew is within the preset window on the screen and the image plane isagain perpendicular to the viewing direction (not shown). The processorin the camera (or a remote processor in communication with the camera)calculates the new distance d_(obnew) of the object to the camera.

The GPS coordinates (x_(c),y_(c),z_(c)) in one embodiment of the presentinvention remain the same. It is contemplated that the camera moves withthe user. This still allows for an accurate determination of thecoordinates. The compass orientation or azimuth of the camera pointedtowards the object is known and the distance to the object is known.This allows the new GPS coordinates of the object to be calculated as(x_(obn),y_(obn),z_(obn)). Based on the new GPS coordinates of theobject and the known coordinates of the platform based camera an azimuthof pointing the platform based camera to the object is calculated andthe actuator(s) of the platform are activated to rotate the platforminto the correct pointing direction towards the object.

In accordance with one aspect of the present invention the hand heldcamera is provided with GPS coordinates of a known track, for instance atrack that is defined by its GPS coordinates on a map. In one embodimentof the present invention a map is provided on a screen of a computingdevice and a track 2600 is highlighted which may be along a road or apath as illustrated in FIGS. 26 and 27. An actual road or path is notrequired as long as points on the track are identified and are providedwith GPS coordinates. The track is either identified on the screen ofthe hand held camera or on another computer device and the track isloaded into a memory on the hand held camera.

The user of the hand held camera has the track 2600 in view from aposition (x_(c),y_(c),z_(c)) and an application is activated on theprocessor of the camera. For instance, the hand held camera 2702 ispointed at a point on the track. The processor of the camera uses theazimuth 1 of the camera 2702 and its GPS position (x_(c),y_(c),z_(c)) tocalculate the intersection of its viewing direction with track 2600. Itdetermines the GPS coordinates of the intersection point which in thisexample is GPS pos. 1. The GPS coordinates of this position areassociated with a viewing direction of platform based camera 2700 withits own known GPS coordinates and the related required azimuth of thecamera 2700 which will activate the actuators of the platform to rotateand point camera 2700 to GPS pos. 1 of the track. In that case camera2700 tracks camera 2702 potentially without having a moving object totrack. Camera 2702 can be pointed to a point on the track with GPScoordinates GPS pos. 3 with an azimuth 3 angle forcing the platformcamera 2700 to rotate also to this point. Camera 2700 may be a handheldcamera, wherein signals including audio, visual, vibratory or touchsignals direct the user to the correct direction.

While for coordinates the term GPS coordinates are used it is to beunderstood that any world coordinates related to a 3D location and thatare determined by a geo-spatial location system can be used.

It is contemplated that the pre-programmed approach also works in a 3Dspace. For instance the hand held or user operated camera 2802 asillustrated in FIG. 28 is in a position above an open space whereobjects or people are moving. The space is calibrated so that allcoordinates inside the space are known to both cameras or to a systemconnected to both cameras. The calibrated space includes at least afloor of the space and a predetermined space above the floor, forinstance a space up to 9 feet above the floor. An operator whichoperates the hand held or operator operated camera 2802 at an object ina location “GPS object.” The orientation or pose and the location ofcamera 2802 determine the intersection of the viewing ray and the floorof the space and thus the “GPS object” coordinates of the object. Thisallows a platform camera 2800 to be directed by actuator into thecorrect direction to have a viewing ray also intersect with “GPSobject.” This allows having a camera 2802 to track a moving object on afloor in a space and have one or more cameras 2800 with a different posethan 2802 also track the object by following directions of 2802.

The object has a size or height. In accordance with an aspect of thepresent invention, a display related to the handheld camera displays theplane of interest. This may be the floor, or the plane wherein a head ofa person is located. This plane can be adjusted by the user. The camerain one embodiment of the present invention detects and segments anobject in the field of vision of the camera. This object can be manuallyselected by the user or automatically detected by the camera in a windowof interest. The bottom of the object or feet of a person are of courseon the floor of the space. This allows the processor to determine theheight of the object and thus the plane or interest for the head of aperson or top of an object. A transparent artificial plane on thedisplay indicates a top of the object. The object image preferably fitsinside a search window.

One can also reverse the roles of the platform camera and the hand heldcamera. A platform camera is tracking an object and a processor alerts ahuman operated camera to track the same object or person. The platformbased camera provides the geo-spatial coordinates of the object to theother camera. The other camera knows its own coordinates and with thereceived object coordinates can calculate the desired pose of the camerato locate the object. Indicators as illustrated in FIG. 6 guides theuser to place the user operated camera in the correct pose to direct itto the object.

In one embodiment of the present invention the object is provided with aposition locator like a GPS device. The position of the GPS device istransmitted to the hand held camera allowing a user to direct the handheld camera to be directed in the correct pose to the object with theGPS device. The handheld camera provides the GPS coordinates to theplatform camera, instructing the platform camera to point at the object.

The calculations may be performed by a processor on the handheld cameraand the processor provides actuator instructions to the platform camera.Desired coordinates are provided to the platform camera, allowing aprocessor on the platform camera to calculate the desired pose and togenerate instructions to its actuator to point the camera in theplatform towards the object.

An object inside an area in the 2D image space of the handheld camera ismarked, for instance with an oval to identify at least part of theobject. The processor segments the identified part of the object fromthe background or extracts features from the object such as edges orcorners or performs any other of known image processing steps, includingdetermining a histogram of pixel intensities and the like. The processorthen marks the center of the object with a blob that may be semitransparent. A user can thus track the object by making sure the blobstays inside the window on the screen by appropriately changing the poseof the handheld camera. A user may place a blob over the object, forcingthe processor to track the object.

The advantage of using blobs is that it is not required that the twocameras “see” the same side of a part of an object. Two cameras may evenface each other in a pose with the object between them. This isillustrated in FIG. 29. It illustrates a screen 2901 on a camera orrelated to the camera that manually tracks an object and keeps an image2903 of the object inside a window 2902. The processor of the cameracalculates a blob 2904 of the image 2903, for instance a circle blobwith a defined radius that falls inside the window 2902 and a center ofthe blob that coincides with the center of gravity of the image 2903.The image 2903 in one embodiment of the present invention is the shapethat tracks the object, such as the earlier mentioned ellipse. Thecalculation of the blob provides an indication to the user to place thecamera in a pose as to keep the blob inside the window 2902. Screen 2901is illustrated at time t=t1.

The screen 2901 also displays an image 2905 of the object taken by thesecond camera on a platform with actuators controlled by signals thatoriginate or are initiated by actions from the processor in the firstcamera. Images may be displayed as opaque or as transparent. Aprocessor, which may be the processor of the first camera but also theprocessor on the second camera, calculates a blob 2906 with a size thatallows it to fit inside window 2902. In the diagram of FIG. 29, image2903 and image 2905 are shown as different shapes. That is because thefirst and second cameras in general observe the object in a differentpose creating different projections of the object in 3D space on the 2Dspace of the respective cameras.

Screen 2911 is the screen of the manually tracking camera shown as 2901at a later moment t=t2. The screen has the same window 2902. It showsalso images 2903 of the object recorded by the first camera and image2905 taken by the second camera. The user has kept image 2903 inside thewindow and new blobs 2908 and 2909 have been calculated. A processor forinstance on the first or the second camera compares the positions of theblobs in a common 2D space. A common 2D space is the screen space of thefirst camera, wherein the screen space of the second camera has beencalibrated against the screen space of the first camera. That means thatthe screen space or the calibrated 2D projective space of the secondcamera is similar or identical to the 2D projective space of the firstcamera. One consequence is that when an object is centered in 2D spaceof the second camera it is also centered in 2D space of the firstcamera. That means that minimizing the distance between the blobs of theobject in the 2D space of the first camera means that both cameras arein a pose towards the object.

Because the pose of both cameras may have changed during tracking theblobs may have become different at time t=t2 which is indicated by thechanged numerals of the blobs.

In a further embodiment of the present invention the first camera whichwas called the handheld camera is also provided on a platform withactuators. A user points the first camera through actuators at theobject and marks the object for tracking by the first camera. Theprocessor in the first camera or related to the first camera then takesover and tracks the object, for instance by creating a blob and keepingthe blob inside the window or inside a predefined area in the 2Dprojective space of the first camera. The processor then tracks theobject with the first camera and collaborates with the second camera asdescribed above.

In one embodiment of the present invention the camera has access to amap of its environment, either through a network or by having a geo-mapstored in its memory. In one embodiment of the present invention thescreen of the camera is placed in one of two modes a camera viewer mode,showing in real-time what the camera is recording and in a map mode,showing a map of the environment based on the location of the camera(determined by GPS geo-spatial coordinates) and the pose, including thecompass direction of the camera. In one embodiment of the presentinvention both the viewer and map screen are displayed on the screendisplay concurrently, either next to each other as illustrated in FIG.30 or overlaying each other wherein at least one mode is transparent. Auser can also switch between the two modes.

In the viewer mode the screen displays in an illustrative example threeobjects 3001, 3002 and 3003 that are viewed in one line. The map modeshows on a map where the objects are located as each object isassociated with a set of geo-spatial coordinates, respectively by:coord_(obj1), coord_(obj2), and coord_(obj3). The coordinates O_(camera)and the pointing direction or compass orientation θ_(camera) of thecamera is also known. A user can see on the map what the orientation ofthe camera relative to the objects is.

Each time the location of the object is determined a new sweeping radiusis determined and provided to the second camera and a new iterativecycle to align the images of the object inside the window or 2D imagespace of the first camera allowing the second camera to track the objectas it is being tracked by the first camera.

Pose determination using sensors such as GPS sensors, digital compass,accelerometers and gyroscopes are aspect of the present invention. Inthe absence of these sensors or in addition to these sensors adifference in pose between two cameras is determined by applying knownimage processing methods.

In one embodiment of the present invention each camera draws a box, anellipse or any other closed contour around the object as is currentlyknown in cameras. A user may place the contour around the object or partof the object to instruct the processor of the camera to track that partor object.

An orientation of a camera has been established. A relative orientationof the cameras can be established by determining the relative positionor angle between the cameras by analyzing edges or lines by applyingprojective geometry as is known to one of ordinary skill. Edges do notneed to be straight lines. However, one can also extract corners of anobject and calculate a line through corners. Furthermore, lineestimating techniques like the Hough transform can also be applied. Theprocessor can be initialized for a known maximum angle between the twocameras. The azimuth is known of camera by the digital compass. Using aprevious setting, this allows an actuator to follow the movement ofanother camera by minimizing an error angle between the two cameras thatis calculated based on the information as explained earlier above.

In accordance with an aspect of the present invention the platform asdisclosed earlier with a housing contains a digital compass to providedata to a processor in a camera with also a digital compass.

Accordingly, methods and systems have been provided that allow a movableplatform with an actuator and a holder holding a first camera to trackan object and record an image of the object based on a second cameralocated on a different spot than the first camera that tracks theobject. The first camera in fact tracks the second camera.

In one embodiment of the present invention a path is read from a digitalmap or recorded, for instance by a GPS device that follows the path. Adevice that is enabled to determine GPS coordinates (including anelevation) is taken to a location. The GPS coordinates are recorded on astorage device, which may be part of a camera as provided herein. Therecorded coordinates may be used to create a map. For instance, GPSpositions as illustrated in FIG. 27 were first recorded with a devicesuch as a camera and then applied to create a map. A map in this contextis a line, which may be a curved line, a straight line or a jagged lineof GPS positions or a contour of positions. The line or contour in oneembodiment of the present invention is created by interpolation of afinite number of recorded GPS locations. For instance the line 2710 canbe created by interpolating and extrapolating recorded GPS positions.

The line 2710 with its related GPS positions or at least theintermediate positions that allow to calculate the line 2710 are storedin a memory, for instance on a server on a network or on a camera. Thepath 2710 may be a path at which a moving object or person is expectedto be present or move. Such a path may be for instance a road over whichrunners are expected or a path going up on a podium, past a receivingline and down from a podium. This allows an object or person to betracked along the path by a camera and a second camera without requiringthe object or person carrying a GPS device. This is illustrated in FIG.31. The path is available either on a server or on a camera 3100 or evena second camera 3102. The position of camera 3100 is known asGPS_(camera) and has a compass orientation θ_(camera). A processor candetermine the GPS coordinates of the viewing direction from 3100 towardsline 2710 as intersection point 3104. This allows the camera 3102 to bedirected towards 3104. For instance camera 3100 is directed towards 3104when a known person or object is at position 3104 at time stamp t1.

In a further embodiment of the present invention a calibrated camera3105 which may be a static camera is recording the path. All images areprovided with timestamps that are preferably synchronized. All imagesrecorded by the calibrated camera allows for determining and marking 3Dlocations in the real world as 2D markings in the image space of therecorded image. One can thus mark with a square in the image recorded by3105 a 3D location such as point 3104. This is illustrated in FIG. 32which shows a video frame 3200 of an image recorded by calibrated camera3105.

The video images recorded by 3105 are recorded and stored on a serverthat is accessible for instance via the Internet. By providing theserver the timestamp t1 and the coordinates 3104 or equivalent data theframe 3200 can be retrieved by the server and a box 3204 can besuperimposed by the server to mark the location 3104 at time “time stampt1.”

In a further embodiment of the present invention the image of object orperson inside box 3204 is marked and is tracked inside the viewing angleof camera 3105 by the processor for the duration that the object isinside the viewing angle of camera 3105.

In one embodiment of the present invention a camera records with animage the GPS coordinates GPS_(camera) (which include an elevation orheight of the camera) and its azimuth or compass orientation. With thecurrent high accuracy location capabilities, it allows the user of anycamera with the same capabilities to assume the camera pose towards anobject at a different time. This is illustrated in FIG. 33. The objecthas GPS or location coordinates GPS_(object). In one embodiment of thepresent invention the object coordinates are acquired, for instance froma map, from another source that publishes coordinates of an object, orby determining the coordinates of the object by taking a picture of theobject with the camera from two different locations by standardtriangulation. For instance by moving the camera so that a rotation ofpreferably 5 degrees relative to the object is realized or morepreferably at least 15 degrees or even more preferably of at least 20degrees, and recording the new camera coordinates and azimuth allows aprocessor for instance on the camera or on a server to calculate thecoordinates of an object. In one embodiment an owner or manager of anobject or a person publishes its geospatial coordinates via a serverthrough a network to a camera. Applying methods described above, thecamera on a screen can provide a user with instructions to direct thecamera to the correct coordinates of the objects. If the camera is aplatform camera with actuators, a processor activates the actuators todirect the camera.

For simplicity, the term GPS or geospatial coordinates is used herein.It is noted that cameras are often rotated in one additional plane to bedirected to a detail of a large object as part of a camera pose. Thisplane may be called the pitch plane and the pitch pose is relative to agravitational center of the earth or to the horizon. The angle isindicated as φ_(pitch) and can be detected with accelerometers orgyroscopes on the camera as explained earlier. FIG. 34 illustrates asituation wherein a detail of an object (a top) is determined by itsgeospatial coordinates GPS_(object-detail). A camera at GPS_(camera) hasto be directed with azimuth θ_(camera) and pitch φ_(pitch) to bedirected at the object detail.

Next to the pitch and azimuth the camera may also have a roll angle asillustrated in FIG. 35. One would in general prefer to take pictureswith a camera that is aligned with the horizon. If not aligned with thehorizon it is assumed that two cameras are at least aligned with theirroll angle. This can easily be achieved by drawing a horizontal and/orvertical alignment line on the screen allowing users to align theircameras.

The term signal is used herein. Signals are processed by circuitry andrepresent data in digital systems. In most cases the term signal orsignals can be replaced by the term data which is processed by aprocessor.

FIG. 36 illustrates for the 2D case how to calculate the coordinates ofan object (xo,yo) when the coordinates and the orientation of twocameras directed towards that object are known. The origin of thecoordinates is placed in one of the cameras and the coordinates arerotated so that the second camera is now located on the x-axis. Suchcoordinate transformation is well known. When angles are known thetangent of these angles are also known. Because the coordinates of thecameras are known the distance between the cameras and thus x1 is known.from that it is easy to determine the coordinates (xo,yo) which in therotated system will be:

xo=x1*(d1/(d1+d2)) and yo=x1*(d1*d2/(d1+d2)) with d1 and d2 the tangentof the angles as illustrated in FIG. 36. It is easy for one of ordinaryskill to expand this to the 3D case and to program this calculation on aprogram for a processor. Additional factors such as elevation andpotentially lens distortions are also taken into consideration.

FIG. 37 illustrates the calculation of an intersection point when thegeospatial coordinates of the camera and an orientation are known aswell as a line or contour from which an intersection point is to becalculated. In this simple 2D example the intersection point (withorigin in the camera) is x0=(b/(c−a)) and yo=ab/(c−a), which can bemodified for the 3D case and programmed on a processor.

In one embodiment of the present invention the platform camera isattached to a moving structure as illustrated in FIG. 38. For instancethe structure is an aircraft on a significantly different elevation thanthe first camera. This requires to have a GPS device and other devicesthat determine the geospatial coordinates of the aircraft. An aircraftdoes not have to rely only on GPS data, but also has other instrumentsthat allow a quick and accurate determination of its position relativeto earth or to a beacon.

Determining geospatial coordinates from other acquired coordinates forinstance in geographic information systems (GIS systems) is known and isfor instance disclosed in U.S. Pat. No. 8,428,341 to Seongho Lee on Apr.23, 2013, U.S. Pat. No. 5,633,946 to Lachinski on May 27, 1997, U.S.Pat. No. 8,265,871 to Starns on Sep. 11, 2012 and US Patent ApplicationPublication Ser. No. 2012/0314068 to Schultz on Dec. 13, 2012 which areall incorporated herein by reference.

The object in accordance with an aspect of the present invention is astatic object. In accordance with an aspect of the present invention theobject has a speed that is less than 1 m/sec, more preferably the speedis at least 0.5 meter/sec., even more preferably the object speed is atleast 1 meter/sec., even more preferably the object speed is at least 5meter/sec. and even more preferably the object speed is at least 10meter/sec.

In accordance with an aspect of the present invention at least onecamera is rotated with an angular speed of less than 1 degree/sec. Inaccordance with an aspect of the present invention at least one camerais rotated with an angular speed of more than 1 degree/sec; inaccordance with an aspect of the present invention with at least anangular speed of less than 30 degrees/min; in accordance with an aspectof the present invention with an angular speed of more than 30degrees/min.

The terms real-time and substantially real-time are used herein.Real-time display of an image means at the same time or substantiallythe same time that an image is recorded and is made available by asensor of the camera.

In one embodiment of the present invention the hand held camera is acalibrated stereo camera which is enabled to determine a distance to anobject. Such cameras are known and are for instance disclosed in USPatent Application Publication Ser. No. 20120327189 to Muramatsu on Dec.27, 20102 and U.S. Pat. No. 8,355,627 to Pace et al. on Jan. 15, 2013which are both incorporated herein by reference. Such stereo cameras,also provided with GPS coordinate determining devices and digitalcompass and pose determining devices make it easier to determine thegeospatial coordinates of an object in view of the stereo camera.

A camera in one embodiment is a smart phone such as an Apple® iPhone® ora tablet such as an Apple® iPAD®. A platform camera in one embodiment ofthe present invention is a distributed system wherein camera,controllers, platform housing, processor, communication systems,actuators and other relevant devices are part of the platform or of thecamera and in some cases both.

In one embodiment of the present invention, a camera is attached to orintegrated into a wearable computer in a head frame. Such head framesare known and are for instance disclosed by Google® as Google GLASS andas Google GLASS Explorer Edition and are disclosed on<www.google.com/glass/start>.

A diagram of a similar headframe 3900 for performing steps of objecttracking provided herein in accordance with one or more steps of thepresent invention is provided in FIG. 39. The headframe 3900 can be wornby a user 3901 as a set of glasses. The headframe includes a camera 3903which in one embodiment of the present invention is forward looking, butcan be directed in any viable direction, a display 3904 for the user3901 and system equipment 3902. Equipment 3902 includes a processor,memory, communication equipment, an antenna, pose determining deviceslike digital compass, accelerometers and/or gyroscopes and GPS or highaccuracy geospatial coordinates determining equipment and a powersource. The frame forms an integrated system that is enabled to performinstructions and to send and receive data to and from a network like theInternet, WiFi networks or a cellular phone network or a small networklike a Bluetooth network for instance in a wireless or wired manner.Data entry devices such as a microphone, a touch pad and a remotecontroller are also provided.

In one embodiment of the present invention the headframe is providedwith a second camera 4001 and a second display 4002.

In one embodiment of the present invention a camera, either a hand heldcamera or a headframe camera, or the headframe holding a camera isprovided with a ranging device 4003 to determine a distance to an objectas illustrated in FIG. 40.

In one embodiment of the present invention a camera in the headframe isa platform camera 4101 that can be actuated to be moved relative to theheadframe as illustrated in FIG. 41.

Accordingly, a first user wearing the headframe has the geospatialcoordinates and the pose of the headframe and camera determined. In onestep of the present invention the pose of the camera and the pose of theheadframe are calibrated. A distance to the object may be determinedwith the device 4003 which determines the geospatial coordinates of theobject or person that is being tracked. A second camera is a differentpose than the headframe camera may also be directed on the object insuch a way that the object image created by the first camera and by thesecond camera displayed on display 3904 overlap inside a window,allowing the geospatial coordinates of the object to be determined. Whenthe object moves it will be tracked by the headframe camera and inaccordance with methods describes earlier above, the object is alsobeing tracked by the second camera.

In some cases it may be required for the user of the headframe camera todirect the second camera to the object. In accordance with variousaspects of the present invention this can be done in several ways. Itcan be done by voice by touchpad or by eye-winks or eye-movementsdetected by a third camera (not shown) in the headframe focused on aneye of the user. Or directions are provided by head movements of theuser that may be quick head movements in the direction wherein thesecond camera has to be moved. In one embodiment of the presentinvention a remote controller 4200 that may be wireless is used toprovide instructions. For instance the controller 4200 has a centralcontrol 4202 that can be pushed or otherwise touched to providedirections which are transmitted via a wireless transmitter 4201 incollaboration with the headframe system. One part of the control 4202has a tactile detectable different form from the other parts (forinstance a bottom part). This allows a user to find the correctorientation of the control without having to look at the control.

The control 4200 in one embodiment has a control button that suspendsthe active tracking of the object when activated and stores the lasttracking data in a memory, for instance by pushing 4202 for a period oftime or by activating a button 4205 or by any other means such as touch,a head movement or a sound or voice command, when the headframe includesa microphone. This allows the user to turn the head with the headframeor remove the headframe without confusing the tracking system. A restartcommand is provided to retrieve the last known position and indicatorson the display will guide the user's head to the right pose andorientation. When restarting the tracking the camera in the headframe inone embodiment of the present invention gets up to date data from thesecond camera if that camera was successful in tracking the object.

In one embodiment of the present invention the first camera loses trackof the object because it disappears behind a barrier, another object, apart of a building, but at least another camera continues to track theobject. According an object is tracked by a first and a second cameraand the first camera loses track of the object, but the second cameracontinues to track the object. In that case the second camera continuesto provide its image of the object to the first camera. In oneembodiment of the present invention the second camera (in fact itsprocessor) continues to determine the geospatial coordinates of theobject and transfers these coordinates to the first camera. The firstcamera may determine that tracking has been lost, because for instancethe known contour of the object have disappeared, or a user for instanceusing control 4200 provides a signal to the processor of the firstcamera that the object has been lost. In that case the processor of thefirst camera shows on its screen 4300 as illustrated in FIG. 43 whichmay be a screen in the head frame or a separate display or a screen on ahand held camera a shape 4301, either with or without a label whichcontains identifying data which may include time stamps and geospatialstamps, that identifies the object as tracked by the second camera. Thisthen appears as shape 4301 on a moment t1 and as a shape 4302 on thescreen as the object is moving behind the barrier and is still beingtracked by the second camera.

The object shape of the hidden object may be represented as a specialicon or an oval or rectangle or any appropriate indicator on the screen.Preferably some color and/or transparency or another property isprovided to the shape indicating that it is hidden from direct sight.Preferably the size of the shape changes with the distance of the objectrelative to the first camera. The shape may also be the image of theobject or person as recorded by a remote camera, and preferably adjustedfor size.

It is noted that a barrier may be a physical object or objects like awall, a fence or trees. However a barrier can also be people. The firstcamera may lose track of a person because he is hidden by a group ofpeople, but is still being tracked by one or more other cameras. In thatcase the tracked person is made visible on the display of first cameraas a shape that is being tracked. In a further embodiment the image ofthe barrier such as the group of people may be displayed in atransparent manner.

In one embodiment of the present invention the user of the first cameracontinues to track the shape of the object by keeping it inside a windowon the screen. The diagram of FIG. 43 in one embodiment of the presentinvention is not preferred as the first camera remains focused on onepoint while the shape of the object moves over the screen.

In one embodiment of the present invention at least a third cameraremote from the user camera and the second (remote) camera is alsotracking the object or person. The at least third camera is positionedin such a manner that during a period of time of preferably at least 1second, more preferably at least 5 seconds, even more preferably atleast 10 seconds, even more preferably at least 30 seconds, and evenmore preferably at least 1 minute, at least two of the at least threecameras are tracking the object in real-time.

It is again pointed out that the term camera herein is a digital camera,with at least one lens, an image sensor, memory, a processor, a screenor display and communication equipment. Additional equipment associatedwith the camera are pose determining devices such as a compass, anaccelerometer, a gyroscope and high accuracy GPS or spatial coordinatedetermining equipment. A camera may also include a loudspeaker and amicrophone and a distance ranging device. All equipment may be part of asingle body or housing or it may be part of several housings wherein theprocessor has access to all devices through internal networking.Furthermore, the processor of the camera is connected with an externalnetwork to exchange data with other devices, such as cameras over anexternal network or over a communication channel between the cameraprocessor and the external device. Unless stated differently, a cameraherein is a camera that includes and/or has use and access to all thesedevices.

A camera can be fixed or portable. A platform may have actuators thatcan be selectively activated by a processor, which is the own cameraprocessor or by an external signal, in preferably at least onerotational plane or in preferably at least two rotational planes, or inpreferably three rotational planes.

The steps and processing of signals are performed by a processor 4401 asillustrated in FIG. 44. Instructions a data to the processor areprovided via an input from a data storage or memory device 4402. Datagenerated by the processor is provided to the memory/storage 4402. Datacan be locally read from the memory via an output. Data and instructionscan be provided to the memory via an input. The processor can receiveinput signals on an input 4410 and can provide for instance controlsignals on an output 4411. Connection of the processor with a network ora wireless channel is provided by a wired/wireless communication device4403 that transmits data from the processor as well as provides data tothe processor 4401. Not shown but assumed is at least one camera incontact with the processor.

As explained herein above, it is beneficial if one can identify alocation of an object or a person, for instance for tracking that personor object with a camera. It is also beneficial if one can remotelyidentify a person or object and/or to obtain information withouttouching the object or directly physically interacting with a person.For instance, in a large crowd it is often difficult to identify alocation of a person, even if one can call that person on a mobilephone. In accordance with an aspect of the present invention a firstperson or object wears or carries a devices that accurately for instancewithin preferably a range of 1 meter, more preferably within a range of50 cm and most preferably within a range of 10 cm determines geospatialor GPS coordinates of that device and thus of the person or object thatwears or carries it.

In one embodiment of the present invention the geospatial coordinatedevice on an object or person is a stand-alone device as illustrated inFIG. 45. The device in one embodiment of the present invention has ahousing 4500 with an antenna 4501 for receiving and transmitting and abattery for power. It has circuitry 4503, for instance including theearlier mentioned BCM4752 with Multi-Constellation acquisition withsub-meter accuracy and fast acquisition marketed by Broadcom® of Irvine,Calif. Circuit 4503 in one embodiment of the present invention alsoincludes pose determining circuitry such as accelerometers, gyroscopesand a digital compass. A user may decide to allow 4502 sending thegeospatial coordinates determined by 4503 to the outside world. A usermay also instruct the device not to transmit that information or onlyselectively to pre-selected targets or in response to pre-authorizedrequesters which may be stored in the memory 4504. The device inside4500 may be controlled wirelessly through another uniquely authorizeddevice like a smart phone or the headframe system as disclosed herein.

In one embodiment of the present invention the geospatial coordinatesare transmitted in combination with an identifying code which may beretrieved from memory 4504. In one embodiment the device of FIG. 45transmits its position on a regular basis to a server in a network or aprocessor in direct connection server, which may selectively transmit itto another device in the network. In one embodiment of the presentinvention, the device in FIG. 45 transmits position data to a server ora device based on a request that may be vetted by the processor in 4502to assure that the requester is authorized to receive the information.

In one embodiment of the present invention the device illustrated inFIG. 45 is a stand-alone device. In one embodiment of the presentinvention the device of FIG. 45 is part of or even integrated into acamera as disclosed herein, in a smart phone, in a computing tablet, ina RFID device, or in a headframe system as illustrated in FIGS. 39-41 orany other computing device, mobile or static that is enabled to performthe tasks of the device of FIG. 45.

A device as illustrated in FIG. 45 may be connected to a network 4600 asillustrated in FIG. 46, which may be a wireless network, an Internet, aswitched network, a WiFi network, a local area network, a micro network,or any other network that allows the device to connect to another devicepreferably wirelessly. A device 4500 as illustrated in FIG. 45 may alsoconnect directly wirelessly to another device, for instance to a mobileor static computing device 4607. Other devices that may be connectedpreferably wirelessly to a network 4600 are for instance a camera 4601,a smart phone, mobile phone or tablet computer 4602, a headframe device4603 as described herein, any computing device 4605, and/or a server4607. All of the devices on the network are enabled to receive and tosend data. The server 4607 in particular is enabled to receivegeospatial end identifying data from 4500 and transmit it to any of theother devices. A tablet computer is a mobile, portable and wirelesscomputer device, usually with a touch screen and commonly larger than asmartphone. A well known example of a tablet computer is the iPad®marketed by Apple Inc. of Cupertino, Calif.

The memory 4504 in one embodiment of the present invention also holdsadditional data that can selectively be sent to requesters or theserver. Such information can be a message, such as a text, audio orvideo message. It may also be a URL referring to a website with forinstance a background or resume of the wearer of 4500, or relevant datarelated to a content or packaging if 4500 is attached to a box or acontainer.

FIG. 47 illustrates this “where are you?” method provided in accordancewith one or more aspects of the present invention. A camera, forinstance operated by a user, but it also may be a platform camera,itself or a connected computing device such as a smart phone puts out arequest for the location of an object ‘obj’. Such a request may forinstance be transmitted over a network to a server. The server matchesthe name ‘obj’ with an ID of a corresponding device as provided in FIG.47 and may send a request for geospatial coordinates to that device. Inone embodiment the device automatically updates its position to theserver. The server checks an authorization to provide that informationto the requesting party.

In one embodiment of the present invention the “requester” is also aportable and mobile smart phone. In that case a user has to move thesmart phone in accordance with instructions (such as directional arrowson the display) provided after calculations based on geospatialcoordinates to put the smart phone of the “requester” in a correct posewhich includes azimuth and inclination and the geospatial coordinatesincluding an height or altitude to have the camera capture an image ofthe object.

If the requester is authorized one of several things can happen. In anembodiment of the present invention the server combines the geospatialcoordinates of the requester and of the target object ‘obj’ to determinea camera pose that will direct the camera to the object ‘obj.’ Theserver may also transmit geospatial data to the camera or relatedprocessors allowing the processor with the camera to calculate therequired pose.

In a further embodiment of the present invention, the current pose ofthe camera is known and the processor provides instructions how tochange the pose of the camera to move toward the correct direction. Itis repeated that a pose has a pitch angle and an azimuth and possibly aroll angle. Geospatial coordinates include an altitude or at least arelative altitude between target and camera.

FIG. 47 shows a group of objects or people (white circles) among whichone object (identified by a dark circle) has to be identified. In thiscase the camera is above the objects or persons, for instance in a largeaudience of 100s or 1000s of people. Even if the target object is insome way marked (wearing a red hat) in general one would not be ablefrom a distance of 25 m, 50 m or 100 m or more to identify the object.

In one embodiment of the present invention the housing of the headframecontains a plurality of vibratory devices, which may be micromotors,buzzers or the like. The vibratory devices are controlled devices thatcan be switched on and off at different times or can be operated atdifferent vibratory frequencies. By activating the vibratory devices ina predetermined order a wearer experiences sensations via the skin thattells which way to turn the head and thus the headframe, which may be upor down and/or left or right.

FIG. 48 illustrates the display of the camera as part of executing the“where are you” method. It is assumed that the camera has receivedinstructions of the pose to be placed in to capture the object. Inscreen 4801, the object ‘obj’ is outside the screen space, which ispreferably a calibrated screen space, of the camera. The object ‘obj’ isshown as the marked circle. The dashes circles are the surroundingobjects to the object ‘obj’. One purpose is to get the object inside thescreen. Another purpose is to get the object inside a window on thescreen. Indicators on the display, herein illustrated as dark arrowsprovide a user with a signal how to turn the camera in the correct pose.The size of the dark arrows indicates how much the camera has to beturned to the right and up to move the object ‘obj’ into the screenspace.

Screen 4802 shows the display at a later moment, wherein the object‘obj’ is now inside the screen space but not yet inside the window.

Screen 4803 shows the object ‘obj’ inside the window. If the object andthe camera are static no further movement of the camera is required andthe arrows turn blank or another indicator is provided that indicatesthat the object is located inside the window. Zoom can be applied toenlarge the image inside the window. Another alert may be triggered ifthe image of the object leaves the window.

However, the camera as well as the object may move, indicated by forinstance the different coordinates of the camera at each screen.

The camera may be a platform camera wherein the actuators areautomatically activated to move the camera in a pose wherein the object‘obj’ is captured inside the window.

The steps of the method are further illustrated in FIG. 49. In step 4902a ‘where are you request’ is generated by a first device with a camerahaving its own geospatial coordinates and pose data. In step 4904 thesecond device provides its geospatial coordinates. In step 4906 thespatial coordinates of the first and second device are used by aprocessor, either in a server or with the camera, to calculate the poseof the camera to focus on the ‘obj’. Instructions to reach that pose arealso provided by a processor in step 4908.

In one embodiment of the present invention a person who will be in acrowd, for instance in a football game audience or in a performance suchas a rock concert, may be provided with an identifying GPS device. Manyevents have large display screens. During game dead-time or to changewhat is shown on the display, often a person in the audience is recordedby a camera and the picture is shown on the large display and/or isbroadcasted. Many people like being shown on a screen during an event.The identifying GPS device can be provided to a person as a favor, forinstance as a reward, or is may be provided as a result of payment witha guarantee of being at least recorded during the event and, possibly ata higher rate, to being shown on a large screen or being broadcast. Onerequirement is that the person carrying the GPS device is within a fieldof vision of a camera, either user operated or automatically directed tothe person carrying the GPS device. In one embodiment of the presentinvention, the carried GPS device has an output device such as a buzzeror screen, alerting the carrier that recording of an image is imminent.A carrier has to acknowledge the alert to confirm that recording isapproved.

In one embodiment of the present invention a smart phone or tabletcomputer has high precision GPS coordinates determination capabilities.In such an embodiment, a person can register on-line for being recordedand installs an authorized application on the device such as smartphone. A smartphone may provide a location of a seat in the venue to aserver, for instance entered by its user. The seat being in a calibratedspace and enabled to be found by a camera on a platform controlled bythe server.

In one embodiment of the present invention a plurality of controlledcameras are installed at a venue such as a stadium. A user of asmartphone or tablet with GPS capabilities signs up for being recordedand is provided a unique identifier (ID). In one embodiment of thepresent invention the person is identified by the ID and is recorded byan overview camera that covers a plurality of people within the camerarange. The user with the ID may be identified by a mark in an image suchas a window that can be selectively removed. In one embodiment of thepresent invention, the person with the ID is recorded in a higherdefinition mode with no or only two or up to 5 or up to 10 surroundingco-attendees. The images may be directly viewable on the smartphone ortablet computer or are stored for later viewing on a server. The imageof the person with the ID may be displayed on a large screen or may bebroadcast.

In one embodiment of the present invention a GPS device with ID is fixedto a body or gear of a person, participating in a performance such as agame like a football game or a dance performance or is fixed in anobject such as a football. In a football game is general an overviewimage is broadcast to an audience and it is difficult to watch aperformance of an individual player. The GPS device that generatesgeospatial coordinates in real-time related to a specific person allowsthe person or object to be tracked by a camera as disclosed herein.These tracking images are transmitted over a dedicated channel to aviewer or to a server.

A person who is the recipient of a “where are you” request may have thecircuit of FIG. 45 in a headframe or has equipment connected with theheadframe. The person may want to know where the requester is located.In that case in a manner similar to the requester the target is informedof the pose (azimuth and elevation of the camera) to locate therequester and is guided as was described above.

The “requester” and the “object” both have communication devices toconnect preferably wirelessly the respective computing devices,preferably via a network that can be wireless, wired or a mix or wiredand wireless connections. However, the devices can also be connected viaa direct connection either wirelessly or via a wired connection.

In one embodiment of the present invention communication between the“requester” and “object” includes a server that for instance performsauthentication, storage and data retrieval and calculations of arequired pose of the requester. In another embodiment of the presentinvention the computing device of the requester receives via acommunication circuit the geospatial coordinates of the object andcalculates a pose to get the object inside a viewing window of a displayrelated to the camera of the “requester.” In yet another embodiment ofthe present invention, the computing device on the “object” requires toreceive via a communication circuit the geospatial coordinates of the“requester.” The computing device of the “object” using also its owngeospatial coordinates and the geospatial coordinates of the “requester”calculates the pose of the camera of the requester that would place the“object” within the field of vision of the camera of the “requester” andtransmits this “pose” to the “requester.” The computing device of the“object” (which may be a person), can deliberately provide someuncertainty, so that the pose determines a range rather than a preciselocation. The actual location of the object may be near a border of awindow of a display at a provided pose and not at the center of awindow. A server, not obligated to the “requester” may also calculate apose for the “requester” without sharing a precise location of the“object.”

In the illustrated example in FIG. 48 a camera is located above theobjects. This is not required. FIG. 50 illustrates in an image 5000 asituation wherein the camera is facing a group of objects or personsforming a barrier to the target object ‘obj’. FIG. 51 illustrates oneembodiment of the present invention in a screen 5100 following steps ofthe ‘where are you’ method and wherein the image of the group of peopleis made transparent and a mark indicates where the target object ‘obj’is located.

The “who are you?” method is to determine with a camera an identity ofan object or a person. Under the “who are you?” method, a person orobject carries a device as illustrated in FIG. 45 and provides a signalwith its geospatial coordinates as well as a public or a secret name,for instance a series of numbers that is meaningless to unauthorizedusers. In a group of objects or persons, each is provided with ageospatial coordinates generating device that transmits the coordinatesand an identifier to a server. The server creates a map which istransmitted to participating cameras which modifies the map relative tothe position of the camera on a display. A user directs the camera,which may be part of a headframe, towards the person or object. In oneembodiment of the present invention the person or object is inside awindow on a display 5200 as illustrated in FIG. 52. The display shows atext ‘obj’ which may be an actual name of the object or a codename.

The requester may point the camera at a target which is shielded byother objects. In that case the requester can change the mode of thedisplay from camera view to map view as illustrated in FIG. 53 asdisplay 5300. The map shows three objects identified as ibj, obj and ebjon the map in the direction of view of the camera. Clearly the objectobj is obscured by ibj and ebj is obscured by obj and ibj. The requesterthen moves the map or a window of the map, for instance by a separatecontrol or head movement or voice with the headframe until theindication or icon of obj on the map is inside the window on the screenand this situation is confirmed. The processor can confirm that anobject inside the window can be identified by changing a color or otherproperty of the window or by highlighting the representation of theobject. At that state the other steps of the ‘who are you’ request asdescribed above are followed.

The objects may not be persons. In that case authorization of therequester is done based on a code that is associated with the requestand that is for instance transmitted by the camera of the requester.

Yet another method that is provided with the cameras as provided hereinis the “where am I” request. This request may apply omnidirectionalimages obtained with a camera of an environment at an earlier time. Suchomnidirectional camera is known and is disclosed in for instance USPatent Application Publication 20010015751 to Geng on Aug. 23, 2001which is incorporated herein by reference. Google operates such a camerafor its Google Street View which is attached to a vehicle driving in anenvironment. The images are associated with geospatial coordinates,allowing a user to view a scene on a computer based on a location on amap.

In one embodiment of the present invention a person wears a headframecamera as described herein above and is located in an area. The wearerwants to know what the environment looked like from a current head poseat an earlier time. For instance, the current environment may be darkand a wearer wants to know where obstacles are. It may also be thatlandmarks have disappeared and a wearer needs to obtain orientationrelative to such (now gone) or perhaps invisible landmarks. It may alsobe that one wants to see the environment under different conditions froma certain perspective.

The “where am I request” works as follows. A camera on a user hasgeospatial coordinates and a pose. Based on the request anomnidirectional image associated with the current geospatial location isselected and provided to the headframe or camera. Based on the pose ofthe camera the display shows only that part of the omnidirectional imagethat corresponds with the pose of the camera, especially the azimuth.The pose in one embodiment of the present invention also includes apitch. Especially when the geospatial coordinates are determined on a cmscale accuracy or better, one can use the headframe camera to walkthrough a darkened or even pitch dark environment with obstacles or anenvironment with covered obstacles and still allow a wearer to avoid allobstacles, or find obstacles by providing previously taken images thatshow the obstacles.

In one embodiment of the present invention directional video can be usedin the “where am I method.” However, that requires that a user isoriented with a camera pose in the direction of the directional video.

The obstacles may be spots on a floor or any locations that should beavoided, but the to be avoided spots, though known by geospatialcoordinates have no visible marks that indicate them. In that case awalk-through video by daylight can be recorded combined with thegeospatial coordinates of a safe path. The safe path can be indicated bya mark shown on the display of the headframe that should be kept insidea window as a user moves forward. The path can be calculated andillustrated as a line or a path drawn on the display to indicate whatthe safe path will be going forwards and where turns have to be takeneven if only coordinates of obstacles are known. One can rotate theheadframe left and rights to see if any spots are marked or of interestoutside the path on the display and to see how wide a safe path is.

In one embodiment of the present invention there may be an offsetbetween the actual geospatial coordinates of the camera and thegeospatial coordinates at which the omnidirectional or directional imagewas recorded. An appropriate 3D-2D transformation corrects the displayedimage for the offset.

An omnidirectional image may be a full spherical field of vision in oneembodiment of the present invention. In one embodiment of the presentinvention an omnidirectional image is 360 degrees in a horizontal planebut limited to a vertical field of vision not greater than 90 degrees.In one embodiment of the present invention the omnidirectional image isin one direction of movement with a horizontal field of vision that maybe smaller than 360 degrees. A field of vision may be smaller than 180degrees or smaller than 100 degrees and is no longer calledomnidirectional.

In one embodiment of the present invention a user held or user worncamera, for instance in a headframe, is provided with a uniqueidentifier that is transmitted to a server on a network as illustratedin FIG. 45. The headframe has a system part, a camera, a display and inaccordance with an aspect of the present invention an identifier part4501.

Several chips (in particular FPGA chips) are available that performreal-time distortion correction in video image. Those chips can beemulated on a processor or can be added as an add-on image processor inthe head-frame. It is noted that aspects of the present invention do notrequire to correct distortion of an entire image taken by a wide-anglelens, but only a part that will be displayed. processing of a smallernumber of pixels will generate results faster at a much higher imagerate, ranging from 1-6, to 6-12, to higher than 12 images per second ofcorrected images that will be displayed on a display.

The Broadcom BCM4752 chip with Multi-Constellation acquisition, whichwas provided earlier above, is applied in one embodiment of the presentinvention to determine a location of a device based on GPS (GlobalPositioning System) signals. As stated earlier, the BCM4752 chipincludes inertial sensors. Inertial sensors in this chip or in otheravailable devices allow the determination of a position, including analtitude, starting from a known position. This is generally known asdead reckoning and is known to one of ordinary skill. Accordingly,positioning systems with chips like the BCM4752 with inertial navigationcapability and with access to GPS data to create an anchor-point enablethe determination of an accurate position, including an altitude notonly when GPS data can be received by the system, but also when GPS datais weak or not directly available, such as inside structures thatsubstantially screen-off GPS satellite signals.

There is an increasing interest in generating accurate indoorpositioning and tracking data, especially as it relates to wirelessdevices such as cellphones, smartphones, tablets and the like in indoorpositions. These indoor positions or locations may be outside asufficient reach of GPS signals. A position of an object may be inside aprivate environment wherein a controlling authority wishes to controland manage location data and/or add specific and potentially proprietarydata to a geo-location request. Several technologies exist that addressindoor positioning, among them GPS or satellite signal derived systems,as described for instance in “Indoor GPS theory & implementation” by RuiXu et al. in Sensors 2015, 15, 10074-10087; doi:10.3390/s150510074, andby pseudo-satellites as described in U.S. Pat. No. 8,791,857 to Mo etal. which are both incorporated herein by reference. Certain methodsapply, what is known as, Assisted GPS or A-GPS, and may applypositioning data from cellphone towers or from local stations orbase-stations and may apply high sensitivity receivers and digitalsignal processing techniques such as described in Indoor GPS TechnologyFrank van Diggelen and Charles Abraham Global Locate, Inc. Presented atCTIA Wireless-Agenda, Dallas, May 2001 which is incorporated herein byreference. IMES is an indoor messaging system that works seamlessly withsatellite navigation systems and is described in “Indoor and OutdoorSeamless Positioning using Indoor Messaging System and GPS” by NaohikoKohtake et al. in 2011 International Conference on Indoor Positioningand Indoor Navigation (IPIN), 21-23 Sep. 2011, Guimarães, Portugal,which is incorporated herein by reference. Other known indoorpositioning technologies apply repeaters, beacons, or stations withknown reference coordinates and are based on WiFi, Bluetooth, NFC, RFID,Ultra Wide Band (UWB), ultrasound, light or other technologies tocalculate a position of a mobile wireless device. Wireless devices mayinclude barometers or other altimeters to determine an altitude.Inertial navigation determination or dead reckoning by a device startingfrom a known position may also accurately determine a position of adevice. An overview of existing indoor wireless positioning technologyis provided in “A Survey of Indoor Positioning Systems for WirelessPersonal Networks” by Yanying Gu et al. in IEEE COMMUNICATIONS SURVEYS &TUTORIALS, VOL. 11, NO. 1, FIRST QUARTER 2009, in “A Survey of IndoorPositioning and Object Locating Systems” by Hakan Koyuncu et al. inUCSNS International Journal of Computer Science and Network Security,VOL. 10 No. 5, May 2010. Circuitry like the earlier provided BroadcomBCM4752 chip with Multi-Constellation acquisition works outside buildingand indoors environment as positioning device.

Many of the indoor positioning systems rely on GPS data as disclosedearlier above. In one embodiment of the present invention this includesa use a known GPS location, followed by a portable device calculatingwith the help of devices such as MEMS, gyroscopes, accelerometers andthe like of an actual position as disclosed earlier above. In oneembodiment of the present invention, certain indoor devices act asvirtual GPS satellites or beacons and allow portable devices byadditional signals related to these virtual satellites or beacons todetermine their relative position related to a known device and theiractual GPS position, if required, related to GPS coordinates.Positioning systems that determine actual GPS coordinates including analtitude of a device are called GPS positioning systems herein. Toindicate that additional processing takes place related to known GPScoordinates, these positioning systems are also called GPS-basedpositioning systems herein.

In one embodiment of the present invention an owner/operator of alocation or positioning system in a building or structure or campus doesnot provide positioning coordinates in relation to a GPS system, butonly in relation to one or more landmarks of the structure, building orcampus. Such a structure or building may be a shopping mall or a campuslike a theme park, an amusement park, or a resort, or a university or aresearch institution or a government campus, or a sports complex, or acruise ship with multiple decks and entertainment areas, or a hospitalor a large warehouse with multiple aisles and stacked storage levels. Alandmark point may be an entrance to a campus or a building, a dedicatedmonument, or a well positioned device relative to which coordinates,often GPS coordinates and including an altitude, are known. Dedicatedmonuments may act as relative origin for positioning coordinates,including an altitude. Those systems that do not rely on GPS data and donot use GPS data are called non-GPS positioning systems. Non-GPSpositioning systems may be indoor as well as outdoor systems. The termGPS herein is used for a system that relies on signals from one or morepositional satellites.

The use of positioning systems on positioning devices, includingcell-phones, smart-phones, tablets and dedicated positioning devices,allows a smartphone or mobile wireless device with a camera to determinea pose of the camera from positioning data, including altitudes,provided by the target positioning device and the smartphone, thatdirects the camera at the target device.

For the definition of pitch herein one is again referred to earlierdescription above and to FIGS. 33, 34 and 35 which shows that the pitchof the camera is the angle of the lens in vertical direction and is alsoknown as tilt or inclination.

Visual instructions on a screen, such as arrows, as illustrated in FIG.48, instruct a user to direct the camera to place an object in a fieldof view. Arrows on a screen with certain length, color or other emphasisindicate how a camera should be moved or rotated to place the object infor instance a center window. A situation wherein a camera 5402 isoutside a walled structure 5400 with an object 5401 is illustrated inFIG. 54, which illustrates a side view. The object 5401 is not directlyvisible by camera 5402 because it is shielded by the structure 5400.Camera 5402 and object 5401 may be in the same horizontal plane, theymay also be in different horizontal planes. The camera 5402 has a posewherein it has to “look up” to point at 5401. FIG. 55 shows structure5400 and camera 5402 in a top view. Device 5401, which is in walledenvironment 5411 is hidden from view of camera 5402 by partialstructures 5410, 5512 and 5514. FIG. 55 also shows an azimuth angle of5402 that determines a pose. The azimuth angle is defined as a compassangle, for instance for a digital compass, in relation to north. Thecamera 5402 is also in a pitch angle relative to a horizon, but this isnot visible in FIG. 55 as it is a top view.

In accordance with an aspect of the present invention, the device thatincludes camera 5402 also is enabled to display a map of its environmentrelative to the position of 5402. The device can switch from camera view(that may show the outside of structure 5400 and a mark that indicatespresence of 5401) and a map view that shows a position of 5401 on a map.In one embodiment of the present invention camera 5402 is an infraredcamera. In one embodiment of the present invention camera 5402 includesa rangefinder which may be a laser range finder.

In one embodiment of the present invention a map displayed to a user of5402 or on a screen of 5402 is a map from a perspective of 5401. Thisallows a user of 5402 to provide instructions to a user on the locationof 5401 to for instance safely leave structure 5400 and it preventsconfusion about taking left and right turns. In one embodiment of thepresent invention the device 5401 also has a camera and a user of 5402can see on a screen which may be a screen of 5402 images generated by acamera on 5401.

In one embodiment of the present invention devices 5402 and 5401 arepre-paired in the sense that device 5401 automatically authorizesrequests from a particular device 5402, for instance because a code sentby 5402 is pre-authorized. In one embodiment of the present invention afirst device such as device 5402 illustrated in FIG. 54 is a smartphoneor a tablet and a second device such as device 5401 illustrated in FIG.54 is also a smartphone or a tablet. The second device with positioningcapabilities, including altitude, may be attached to an object orperson, like a young child, unable to operate the second device. In thecase that the first device is already pre-paired to the second device,the second device still has to receive a request, but recognizes forinstance a code that identifies the first device as authorized andpermits and enables data exchange of positioning data between thedevices. In one embodiment of the present invention the secondpositioning device has no camera and/or certain other capabilities thatstandard smartphones have such as a microphone, or a loudspeaker, or alarge screen. Its main purpose it to generate positioning data,including altitude upon request and to wirelessly exchange data withanother device. In such an embodiment the second device can be quitesmall and attached or part of clothing or attached to clothing or to abody of an object or a person or an animal and has only the requiredcomponents, circuitry, antenna and power source to act as a wirelesslocation/positioning device with communication capabilities and may havea small screen to display status data.

In one embodiment of the present invention the first and the secondpositioning device both receive satellite navigation signals and use acommon GPS or satellite navigation system coordinate system. In oneembodiment of the present invention at least one of the two devices, asillustrated in FIG. 54 for instance, operates in an indoor positioningsystem that operates with its own coordinate system while another deviceoperates in a satellite navigation system coordinate system. Inaccordance with an aspect of the present invention a translation of onecoordinate system into another is enabled. Such enablement may be on oneof the two devices. The coordinate translation may also be provided on aserver in a network that can be contacted over a network as illustratedin FIG. 2.

One may call aspects of the present invention: seeing through walls orseeing through crowds or seeing through objects. In one embodiment ofthe present invention a data file which may include image and/or sounddata is associated with a device such as device 5401. Part or allcontent of the data file related to the object or person 5401 isprovided to the requesting device 5402 where it may be displayed. Boththe requesting device 5402 and the searched device 5401 have theirpositional coordinates, including an altitude, which allows a processorin 5402 to determining a pose that puts 5401 in the field of view of acamera on 5402. In one embodiment of the present invention, the objectis a storage container among a plurality of storage containers that maybe stacked and obscuring each other in a stacked fashion. This isillustrated in FIG. 54 wherein object 5401 with an ID and positionaldevice is inside or attached to a storage container 5411. Container 5411is part of a stack of storage containers 5400 that includes containers5410, 5411, 5415 and 5416. The containers may be boxed, drums, chests,shipping containers that may be stacked in a warehouse, a yard, a ship,a truck or any place else where containers may be stacked. A containerthat is present in a stack and has a positional device can easily befound with a corresponding searching device. Before one starts movingcontainers out of the way to remove the identified container one cancheck on the requesting device the data file that describes a content ofa container.

In one embodiment of the present invention a positional device isattached to a vehicle like a car or a truck. The vehicle may be storedor parked on a site with many vehicles. The positional device in thevehicle is paired with a requesting device like a smartphone. Therequesting device transmits a request to the positional device in thevehicle and the user of the requesting device can find the vehicle withthe requesting device which may be a smartphone. This is useful infinding a vehicle on a parking lot for instance at an airport after atrip and the user may have forgotten a location of a vehicle. A similarproblem may arise with locating a rental vehicle. A rental vehicle maybe parked on a parking lot or on a street and the color, make and exactlocation of the vehicle have been forgotten. An image of the vehicle maybe displayed on the smartphone. The disclosed aspects of the presentinvention are helpful in locating the vehicle. This is safer than onemethod that is often used by activating a door opener which oftenactivates outside lights. However, if one misses finding the vehicle itmay be left with its doors unlocked. In accordance with an aspect of thepresent invention a smartphone of a user is paired or authorized for alocation request with a positional device in or attached to a vehicle byan authority that is authorized to perform such a pairing. Such anauthority may be an authority that authorizes a rental or use of avehicle to a user, for instance for a certain period. In accordance withan aspect of the present invention the authorization of a locationrequest between a user requesting device such as a smartphone and apositional device (for instance on a vehicle) expires after apredetermined period or by an action of the authority and the pairing isdisabled.

A vehicle may be parked in a multi-level parking garage. Forgetting theparking location, especially the level of the parking spot in the garageis very inconvenient. In accordance with an aspect of the presentinvention a location, including the level of the parking spot can easilybe retrieved by applying one or more aspects of the present invention.In one embodiment of the present invention a user determines if thevehicle is located on a higher or lower or at the same level as therequesting device. In one embodiment of the present invention a parkingauthority or management operates a parking specific location systemwherein each parking spot has its specific coordinates which may be GPScoordinates or GPS-based coordinates or may be provided as parkingspecific coordinates (level X, aisle B, spot 7 for example). When avehicle is parked on a spot, its coordinates are translated and/orstored as local coordinates on the positional device of the vehiclebased on a connection with a local parking system. A location requestfrom a smartphone to a positional device will receive as a response thepose instructions but also the local coordinates such as level, aisleand parking slot number.

In one embodiment of the present invention a smartphone with a cameraand positioning circuitry is paired with a positioning device withcommunication capability that authorizes automatically the smartphoneand that is attached to an animal, for instance it is attached to acollar of a dog. Dog and cat owners may experience that their animal,for different reasons, moves away from supervision and/or a knownenvironment. This often leads to anxious situations, because animalsthat feel lost often hide below structures or in bushes or in trees andmay be extremely hard to locate. Aspects of the present invention willbe helpful to locate animals that are hidden, especially in lowlocations such as basements and high such as in trees when they carry apositioning device as provided herein.

In one embodiment of the present invention a presence of a positioningdevice needs to be located (or its absence confirmed) after a locationof an object has been determined. This is in some respects a reverse ofprevious aspects of the present invention where a location is unknown toa searcher but an ID of a searched device is known. Earlier above, itwas shown that a location of an object can be determined by two camerasas illustrated by for instance in FIG. 3. In certain cases one may haveonly access to one device such as a smartphone and there is a need tofind information about a device with location capabilities including analtitude which may be located among other devices with similarcapabilities. For instance a device is attached to a container and thecontainer is stacked and perhaps even obscured by other containers. Onemay be in need of information about a certain container. In oneembodiment of the present invention a device like a smartphone withpositioning capability is directed at a container with a positioningdevice (potentially obscured by another container, also with apositioning device). This is illustrated in FIG. 55. Camera 5402 that ispart of a computing/communication device that can determine a pose andcoordinates (orientation at a location) with devices like MEMS,gyroscopes, accelerometers, digital compass, outdoor GPS and/or indoorpositioning when pointed in a direction of the container. For instancecamera 5402 is pointed at container 5411 in a stack with positioningdevice 5401. Container 5411 is obscured in the stack from the camera5402 by container 5410 with positioning device 5420. One way to find outthe identity of a container at the position of 5401 is to interrogateall containers in the area to ID themselves.

A more efficient way to find out about the containers is find out theidentity of containers that are in the field of view, even whenobstructed, of camera 5402. In accordance with an aspect of the presentinvention the positioning devices 5420 and 5401 are associated withtheir actual position (GPS or indoor coordinates) in a database. Assumethat 5420 presently has horizontal coordinates (x1,y1) and altitude z1(or general coordinates (x1,y1,z1)) and 5401 has coordinates (x2,y2,z2).It is assumed for an embodiment of the present invention that the ID ofthe object such as a container and the correct ID of the correspondingpositional device is unknown or uncertain to a device such as asmartphone with camera or the user thereof. The smartphone is in contactwith a database, which may be stored on the smartphone or on a locallyaccessible device or on a remote database that can all be accessed fromthe smartphone as illustrated for instance in FIG. 2. The challenge forthe smartphone is to determine or at least estimate a location of theobject (coordinates, including an altitude). Based on the determined orestimated coordinates, the database is searched to find the object orobjects that are occupying coordinates that are identical or at leastclose to the determined or estimated coordinates. Only one object canoccupy specific coordinates. To address potential inaccuracies, one mayset an area of uncertainty around the estimated or determinedcoordinates. The database will provide all stored objects that arelocated within that area. In accordance with an aspect of the presentinvention also a time frame is defined that instructs the database toprovide a history of objects that have occupied the area. The problem tobe solved is to determine or estimate a location of an object with thesmartphone with camera of which a location and a pose are known. One ofordinary skill knows how to determine coordinates of a point relative toa camera or smartphone, when the coordinates of the camera or smartphone(xc,yc,zc), additional pose parameters (Θ_(azimuth) and Θ_(pitch)) anddistance D from smartphone/camera to point are known or estimated. Thisis illustrated in FIG. 56 where a distance D from camera/smartphone 5602to a visible point on an object 5610 is measured or estimated. Theobject 5610, which may be a container or a person or an object hasattached or inside it a positional device 5606 with position coordinates(x1,y1,z1) which may be positioned above object 5611 which contains orhas attached a positional device 5607. How to calculate coordinates(xe,ye,ze) of a visible spot on object 5610 based on the available databy standard geometric calculations is known in the literature. It isexplained in for instance Kuscer et al.: Measurement UncertaintyAssessment in Remote Object Geolocation, Journal of MechanicalEngineering 59(2013)1, 32-40, DOI:10.5545/sv-jme.2012.642, which isincorporated herein by reference.

A database as provided above is in one embodiment of the presentinvention a Geographic Information System (GIS) database for instancesuch as marketed by Esri of Redlands, Calif. under brandname ArcGIS foraccessing and processing positioning data and described in URLhttp://desktop.arcgis.com/en/arcmap/10.3/manage-data/geodatabases/types-of-geodatabases.htmwhich is incorporated herein by reference. A database for storinglocation information of shipping containers is disclosed in U.S. Pat.No. 7,194,330 to Robert Carson issued on Mar. 20, 2007 which isincorporated herein by reference.

The determination of the distance of between the camera/smartphone andobject requires a rangefinder or at least a rangefinding method. In caseof objects that have known sizes (such as shipping containers) one mayapply Mil-dot Range Estimation as is known in the art. Furthermore rangefinding applications that apply the camera in a smartphone (and that aregenerally applied in the game of golf). In accordance with an aspect ofthe present invention a dedicated range finder tool 5603 is used by thesmartphone 5602. One rangefinding tool that works with a smartphone is alaser based rangefinder of which an example is manufactured/marketedunder the brand name Ryobi Power Tools marketed by One WorldTechnologies, Inc. of Anderson, S.C. Other types of rangefinders applyradar or ultra-sound. An affordable LIDAR rangefinder that works with asmartphone is provided in “A smartphone-based laser distance sensor foroutdoor environments,” Gao et al., Computer Science and ArtificialIntelligence Laboratory, EECS Department, Massachusetts Institute ofTechnology, downloaded from “URLhttps://projects.csail.mit.edu/wiki/pub/LSPgroup/PublicationList/PhoneLidarICRA2016.pdf”and which is incorporated herein by reference. A sonar based rangefinder is provided in “A Software Based Sonar Ranging Sensor for SmartPhones” by Graham et al., downloaded from URLhttp://www.cs.wm.edu/˜gzhou/files/SonarSensor_iot15.pdf which isincorporated herein by reference. Radar based rangefinders are alsoknown.

A rangefinder may be close to the camera. The rangefinder may also be ashort distance away from the camera. The rangefinder (if a devicedifferent from the camera) is substantially close to the camera, whereinsubstantially close is preferable within a radius of 5 meters, morepreferably within a radius of 2.5 meters and most preferably within aradius less than 1 meter. The distance that is determined between objectand rangefinder is substantially, including variations as providedabove, the distance between camera and object.

Rangefinders have an uncertainty in their results which translates intoan inaccuracy. Furthermore, the distance to a visible outside point isusually determined by rangefinding devices that rely on cameras or“flight-of-time” principle. The actual position of a positioning device5606 may be on the back or rear of a container and may provide avariation of a length or depth of a container. In accordance with anaspect of the present invention an estimated location (xe,ye,ze) iscalculated by the computing device or smartphone based on the availabledata. Based on other data, such as size and density of objects andaccuracy of the calculations and measurements a “location box” 5615 isdetermined. The “location box” is defined as a location search box thatis based on the calculated or estimated coordinates (xe,ye,ze) and withboundaries defined by a preset variation around (xe,ye,ze). In FIG. 56‘location box’ 5615 is illustrated as a cube with symmetric variations±Δx, ±Δy and ±Δz. This is for illustrative purposes only. The presetvariation may be asymmetric, different in each major axis and direction.The “box” may also not be a cube, but can be any shape that isappropriate, including a sphere or an ellipsoid. What is important isthat the boundaries of the ‘box’ are well defined.

In accordance with an aspect of the present invention, a smartphone orcomputing device determines, based on a distance between itself and anobject and a pose that places the object in a field of view of a cameraof the device, a “location volume” with boundaries, the boundaries beingdefined by positional coordinates, which may be GPS coordinates orindoor coordinates. A pose herein is a directional pose determined bytilt or pitch and azimuth or pan direction. In one embodiment of thepresent invention, the device or smartphone has records or a database ofobjects with their spatial or position coordinates. The computing devicesearches its records or database to identify and/or list or displayand/or make available for review all objects that are located within the“location volume.” In the illustrative example of FIG. 56 there are twoobjects with positioning devices in each neighborhood. The camera isclearly pointed at object 5610 and there is for now no interest in 5611.As an illustrative example the “location volume” is set at for instancenot more variation than 30% in the z-direction. Because the locationdevice 5606 may be at the back of object 5610, a preferred variation inthe x-direction may be 150% of the known depth of object 5610. Similarreasons may be applied to the y-direction, if the y-direction is thelength of a container for instance. The database or records search inone illustrative example returns with one positioning device beingidentified within the “location volume.” This indicates that the foundID likely pertains to 5610 and the smartphone is instructed to displaydata related to this device. If no ID is found then the “locationvolume” is extended.

In one embodiment of the present invention it is desirable to limit asize of the “location volume” for instance to limit a number of objectsthat may be found. This is illustrated in FIG. 57 wherein a stack of 3rows and 3 columns of containers is illustrated. A surveyor is aware ofa structure of the stack and is interested in the top row and inparticular the ‘hidden’ container 5730. It would be confusing to getinformation about all containers in the stack. To limit the retrieveddata found in a database and displayed on a screen in the device thatincludes camera 5602 the “location volume” is preset and limited insize, for instance as size 5708. The area outside the “location volume”is called an “exclusion zone.” Both the “location volume” and the“exclusion zone” are based on a size of an object, a number of expectedobjects and an expected accuracy of the determined “location volume” tocapture the object that is searched for. In accordance with an aspect ofthe present invention the “location volume” is adjusted or modifiedbased on a search result. The stack of containers 5700 has containers5710, 5711, 5712, 5720, 5721, 5722, 5730, 5731 and 5732. Assume for thisexample that all containers coordinates and other data are stored in adatabase searchable on positional coordinates of the containers. Allobjects or containers occupy a volume. The positional coordinates of acontainer may be determined by the positional coordinates of apositioning device such as 5740 or by the coordinates of one or moremarks or identifying features of a container or a calculated point likea center of gravity of an empty container. A “location volume” that istoo short may find container 5732 and even 5731 but not 5730. Based onother information a surveyor knows that the stack has at least 3 columnsand by re-sizing (enlarging in depth) of the “location volume” will beable to find 5730. A “location volume” that is too high may alsogenerate 5712, 5711 and 5710 which may be considered as clutter in asearch result. In that case the height of a “location volume” should bediminished. If the location coordinates of an object in a stack are welldefined (such as upper right corner facing the camera or center ofgravity of an object) then a much narrower “location volume” can be usedas such a “location volume” is most likely to capture or cover thelocation coordinates of an object such as a container. This is furtherillustrated in FIG. 57. Assume a pose and coordinates of 5602 and adistance measured by 5603 identifies a point (xe,ye,zm) on container5722 in the second row. Assume that the center of a container determinesits location. If one is only interested in the last object or container5720 in this row then a “location volume” 5709 will lead to a properfind in the database. This is a form of “looking” behind objects byplacing the locations that obscure a particular location in an“exclusion zone.”

An object generated in a database location search may have a positioningdevice which may have a particular ID. When the ID of a device isprovided as a result of a search and there is authorization or pairing,one may apply the earlier above described steps and/or devices to place(a potentially hidden) object in a field of view of a camera.

In accordance with an aspect of the present invention an object is astructure or a part of a structure such as a room, a hospital room, anoffice, an apartment, a cell, a part of a boat and the like. A surveyorwho is operating device 5602 may be a security officer who noticessuspicious activities in a warehouse or another building with rooms orwalled spaces 5710 to 5732 as illustrated in FIG. 57. Assuming that dataof such walled spaces is stored in a coordinate searchable database,steps and apparatus as provided herein can assist in finding moreinformation about a specific space. A positioning device 5740 may bedetected and may be contacted as described earlier.

In one embodiment of the present invention the camera is pointed at anobject like a person, a shipping container, a vehicle, a building or anyobject that acts as a facade that may be opaque and that obscures one ormore objects or rooms or structures in a location volume from view by acamera. In accordance with an aspect of the present invention atransparency of the image of the facade object is increased and objectsidentified to be in the location volume are drawn in a wireframe orcontour by the processor on the display, based on information in thedatabase. Objects may also be identified as blobs, contours or shapessuch as rectangles and ellipse to provide a viewer context oforientation of hidden objects relative to the facade that obscures thelocation volume. This is called “drawing an outline” or “drawing anoutline of a hidden object” or “drawing an outline of an object in thelocation volume.”

This is illustrated in FIG. 58 where on screen 5803 an image of a wall5820 of an object recorded by camera 5804 displayed in a transparentmode. Drawn into the image based on available data are the contours of aroom/space/object 5822 and an ellipse identifying an object 5821 with apositioning device. The processor may also draw contours of a locationvolume, which is not shown herein as to not overcrowd the drawing withtoo many details. A position on a screen of an outline of an object iscommensurate with its positional coordinates and a size of an outline iscommensurate with its actual size in an embodiment of the presentinvention. An outline of the location volume may also be provided on ascreen.

In one embodiment of the present invention an object is a person who isobscuring visibility of another person, for instance in a crowd. Or anobject is an animal, like a horse in a herd of horses and obscuringvisibility of at least one other horse.

In defined environments, including warehouses or container yards,position device readers are placed and facilitated (for instance byad-hoc networks) in such a way that all positioning devices can becommunicated with. Also, cameras may be attached to a movable gantry orarm or a moving or static pole to facilitate viewing of objects by thecamera, like 100 s of new cars in a parking lot, each with their ownpositioning device.

In one embodiment of the present invention the camera is movable as tobe at least rotatable. In one embodiment of the present invention thecamera is movable as to be translatable in the x-y plane. In oneembodiment of the present invention the camera is mobile, portable andmovable such as a camera in a smartphone. In one embodiment of thepresent invention the camera is earthbound as being attached to anearthbound structure which includes a building or a car or truck or aboat or other earthbound vehicle or structure or operated by a person onearth who is not flying. In one embodiment of the present invention thecamera is not earth bound as being attached to an aircraft including anairplane, a helicopter or a drone or a person flying in an aircraft.

Under certain circumstances, it is not desirable to have positioningdevices to be alerted that they are receiving a positioning request. Insome cases a decision is required if an action has to be taken withregard to an object or person in a group. Rather than finding allmembers of the group against who no action is required, it is beneficialto find that in a sphere of action no “no-action” people or objects arepresent which can be achieved by applying aspects of the presentinvention.

In one embodiment of the present invention the searching device withcamera is one of the group consisting of 1) a smartphone; 2) a tabletcomputer; 3) a laptop computer; 4) a mobile computing device in awearable headframe such as illustrated by FIG. 39; 5) an assembledcomputing device including a) a programmed processor with memoryconfigured to perform steps of the present invention b) power source c)communication circuitry d) at least one antenna d) a camera e)connectors or connecting equipment such as Bluetooth wireless circuitryf) a screen; g) a housing h) controlling circuits which may includebuttons and/or touch screen controls or other controls i) a digitalcompass j) GPS circuitry k) circuitry such as accelerometers and/orgyroscope l) an on/off control m) positioning circuitry like theBroadcom BCM4752 chip with Multi-Constellation acquisition and n)range-finding circuitry and/or devices.

In one embodiment of the present invention the camera 5602 is part of asmartphone or tablet computer which has been modified as illustrated inFIG. 58. FIG. 58 is a diagram of a tablet computer or smartphone 5800,which is shown in landscape orientation but is also used in portraitorientation. It has a screen 5803, which in one embodiment is atouchscreen from which instruction to the processor can be provided. Ingeneral a smartphone and tablet have two cameras: a rear camera on theopposite side of the side with the screen of which lens 5802 is shown indashed lines because it is not visible in this view and a front camerawith lens 5801 on the side with the screen. Using the camera with lens5802 for aspects of the present invention is a bit awkward because thetablet or smartphone has to be held up and it is difficult to enterinstruction on the touch screen 5803. It is more convenient to have acamera on an edge of the smartphone or tablet. This is illustrated ascamera 5804 which is attached as an illustrated example on the top edgeof the smartphone or tablet in landscape mode. Also shown is arangefinder 5805 which is also attached to the edge of a body of thedevice 5800. A body of 5800 has a length, a width and a thickness. Theedge of 5800 is the side usually designated as having the thickness. Thecamera 5894 substantially lies in the plane of screen 5803 or parallelto 5803 and is perpendicular to front and rear camera lenses 5801 and5802. The camera 5804 and the rangefinder 5805 may be integrated into abody of the smartphone or tablet 5800. This allows a pre-use, factorydirected, calibration of the camera and rangefinder to integrate withsensors such as pose sensors and GPS sensors in device 5800. The camera5804 and rangefinder 5805 may also be fixed into a harness or holderthat is clamped to device 5800. The holder with camera and rangefindermay have their own controllers and memory and communication circuitrywhich allows it to communicate with a processor in 5800. The holderstructure allows the camera 5804 and/or rangefinder 5805 to be attachedor detached from 5800 and to be used in portrait and landscape mode of5800. This embodiment allows steps of the present invention to beperformed while device 5800 is not blocking a view and also providingannotations or instructions on the screen is greatly facilitated. Theinclusion of a rangefinder 5805 may be dropped if images only are usedto estimate a distance.

Different configurations are used of computing device with camera andoptionally a rangefinder and a pan/tilt motorized platform. When thecamera is attached to the computing device, which includes beingintegrated in the body and clamped to the body, positional dataincluding an altitude of the computing device is intended to mean alsopositional data of the camera. A pose of the computing device isintended to include a pose of the camera. If the camera is attached toan edge then pose data have to be adjusted for a rotation with 90degrees or a multiple thereof. For the configuration of FIG. 60 which isdescribed herein, the positional data including an altitude is the sameas that of the computing device or substantially the same. However,because the camera body 5900 is rotatable independent from the computingdevice, additional devices are required. In one embodiment of thepresent invention device 5900 has its own pose sensors and maycommunicate those to device 6000. In one embodiment of the presentinvention the combination of connector 5903 with receptor 6001 includesan axis position sensor which determines the angle of rotation of 5900relative to 6000 which is used with the angular position of 6000 todetermine the pose of 5900.

In certain cases a device like 5800 has to be held in a certainposition, for instance for convenience or for limited room availabilityor for other reasons. It thus would be beneficial if the position ofcamera and/or rangefinder does not strictly depend on a position ofdevice 5800 and offering a user the benefits of using the screen whileapplying the camera in accordance with one or more aspects of thepresent invention. In accordance with an aspect of the present inventiona camera device 5900 as illustrated in FIGS. 59A and 59B is provided.The camera device 5900 has at least a camera of which lens 5901 is shownand a rangefinder 5902 of which a transmitter/receiver 5902 is shown.The rangefinder may be optional in one embodiment of the presentinvention. The camera device is embodied in a body or housing 5900. Thebody or housing contains circuitry 5904 and optionally a power source.Details of components and circuitry, including connections, in FIGS. 59Aand 59B in 5900 have been omitted as not to obscure a purpose of thedevice, but should be assumed as one of ordinary skill knows. Thecircuitry contains control circuitry, sensors such as pose sensors,communication circuitry to communicate with device 5800, if so desiredseparate positioning or GPS circuitry. Device 5900 may optionally have asmall display to provide status data and thumbnail images. Furthermore,5900 has a connecting device 5903 that enables device 5900 to bemechanically connected to device 5800 via a rotatable mechanicalconnector 5905. FIG. 59A shows a frontal schematic view of device 5900and FIG. 59B shows a side view of 5900. Device 5900 is at leastrotatable 5903 with some friction so that the device 5900 can be placedin a semi-fixed but movable position relative to 5903. Connecting device5903 in one illustrative example has a preferably not circle crosssection. In one embodiment of the present invention a smartphone/tablet5800 is modified to smartphone 6000 that is provided with a receptacle6001 as shown in FIG. 60 to receive 5903 and that firmly but removablyholds 5903. In one embodiment of the present invention connector 5903 isused to provide power to 5900 from a power source in 6000 whenconnected. Circuitry in 5900 determines a pose of 5900 relative to ahorizon. Device 5900 may have its own positioning circuitry, todetermine positional coordinates, including an altitude. Device 5900 mayalso rely on device 6000 or 5800 for positional coordinates. In oneembodiment device can be rotated in a plane perpendicular to and around5903 and thus parallel to the side of 6000. In a one embodiment of thepresent invention 5900 is also movable, because of ball connector 5905in multiple planes. Accordingly, device 5900 can be placed in adesirable pose while computing device 6000 is held in a preferred orsubstantially preferred position. Data is exchanged between 5900 and6000, preferably wirelessly and data and images as well as instructionsto move 5900 in a particular pose can be viewed on screen 5803. In oneembodiment of the present invention device 5900 can be handheld andspatially separate from either 5800 and/or 6000. Device 5900 may stillbe connected for communication with 5800 or 6000, and such communicationis either wireless or through a cable.

In one embodiment of the present invention device 5900 is integratedinto a headframe as illustrated in FIG. 39. In one embodiment of thepresent invention the display 3904 in the headframe is used to viewimages or data. In one embodiment of the present invention the screen3904 displays data from separate device 5900. In one embodiment of thepresent invention device 5900 is integrated into headframe 3900. Howeverthe data is viewed on a separate device 5800 which is in communicationwith 3900. Preferably camera 3903 is rotatably fixed on 3900 so a userof 3900 can look down on device 5800 while the user rotates 3903 in apreferred pose. In such an embodiment a pose of 3903 is derived from apose of 3900 or camera 3903 has its own pose sensors that determine apose of 3903 relative to a horizon.

In one embodiment of the present invention other constructions to attach5900 to a computing device such as 5800 are contemplates, includingclamping and screw constructions. Preferably a construction holds acamera and optionally a rangefinder in a fixed position. An illustrativeembodiment of a motorized platform 6100 is provided in diagram in FIG.61. A camera device 5900 with optional rangefinder and with sensors thatenable determining a pose of the camera is attached to a motorcontrolled platform. The device 5900 is, preferably removably, attachedwith a connector or fixture 6101 to a motor rotatable platform 6102which itself is attached with a motor tilting or motorized hinge likefixture 6103 to a base 6104 that is attachable to a structure like atripod, a frame or other parts of a vehicle or to any other structurethat may be static, moving or movable. The motorized platform asillustrated in FIG. 61 has circuitry 6105 to receive signals to activateand control one or more motors. A power source for the platform isassumed but not shown as well as circuitry to communicate with 5900and/or with a computing device 5800/6000. Some elements are not shown inthis and other drawings to prevent obscuring other details as providedherein. The platform 6100 of FIG. 61 is generally available and is forinstance known as motorized and remote controlled pan/tilt scanner heador pan/tilt camera head and are marketed by companies such as Pelco ofClovis, Calif. and Bescor of Farmingdale, N.Y. In one embodiment of thepresent invention positioning of the camera and optionally therangefinder in 5900 in a desired pose with the platform 6100 in pitch(tilt) and azimuth (pan) is calibrated with the sensors (accelerometers,gyroscope and digital compass) of 5900 and the controls of the platform6100.

As illustrative examples: 1) Based on calculations and data exchange itis determined that a camera should be in a pose of 37.2 degrees tilt and49.5 degrees azimuth (NE). In one embodiment of the present invention acontroller of the platform provides motor control signals that placesthe camera in the pose. The motor control signals may be derived from atranslation table. The motor control signals may also be derived fromminimizing a difference between desired pose data provided from thecalculation and actually measured pose data as provided by the sensors.2) A user wants to place a camera in a desired pose with the motorizedplatform. A menu control, a button, sliders on a touch screen, othercontrols on a touch screen or a joystick is provided to control theplatform. 3) The motorized platform in a further embodiment of thepresent invention is controlled from a mobile computing device 5800 or6000. A pose of the computing device, measured by sensors inside thecomputing device which are separate from sensors in 5900, determine adifference in rotational pose (pan and tilt) of the housing of thecomputing device relative to a neutral position of the housing of device5800 or 6000. The pan and tilt of the motorized platform is changedbased for instance on the change of pan and tilt of the device. In oneembodiment of the present invention the motors of the motorized platformare activated when the computing device 5800 or 6000 is out of itsrotationally neutral point and stop working when the computing device isin its neutral pose with a neutral pan pose and a neutral tilt pose.

The motorized platform allows the camera to be placed and held in apreferred pose even if the base 6104 is attached on a structure that ismoving. The camera is placed in a preferred pose, for instance by auser, from a computing device 5800 or 6000 and the system is instructedto hold that pose. The initial pose is determined from an actualposition of a target or a target location is determined from an actualposition and a determination or estimation of a distance. In bothsituations a set of coordinates for a target, including an altitude, isavailable to the system comprising the computing device 5800 or 6000 and5900 on platform 6100. As the platform moves, it generates newcoordinates of itself for instance by using positional circuitry in5900, including an altitude and the computing device 5800 or 6000calculates a new pose for the camera in 5900 to place or keep a targetin its field of view and instructs the motors in 6100 to adjust panand/or tilt of the platform to place the camera in the correct pose.Based on a rate of change or speed of the platform predictive measuressuch as a Kalman filter or other control devices are used to prevent orat least diminish loss of target of the camera and to address noise-likechanges in the position and pose.

FIG. 57 illustrates an example of a location volume 5709, which hasboundaries that are for example parallel to the ground. Otherorientations and shapes of location volumes are contemplated and may beangled relative to the horizon or of irregular shape. The term horizonis applied herein as to be a common horizon would be understood by oneof ordinary skill in navigation. One may use any of horizons as known,including an artificial horizon as available on smartphones andcomputing tablets. In case of doubt a horizon is defined as “One ofseveral lines or planes used as reference for observation andmeasurement relative to a given location on the surface of the earth,and referred generally to a horizontal direction (i.e., at right anglesto the zenith)” as provided by the American Meteorological Society.However, different horizons may be used, as long as they lead tosubstantially a camera to be positioned in a correct pose with adeviation that is preferably not greater than 15 degrees from an ideal,more preferably a deviation not greater than 10 degrees and mostpreferably a deviation not greater than 5 degrees wherein an ideal is acamera be positioned so that an object is exactly in the center of afield of view.

In one embodiment of the present invention an object is recorded into adatabase by determining, for instance with a smartphone with positioningcircuitry, positional coordinates of an object or a spot in an object orstructure or room. If the structure is movable or mobile, such as avehicle or a shipping container, one or two or more positionalmeasurements of recognizable elements, such as left front and rear lightin a vehicle or upper corners of a shipping container as illustrativeexamples, are recorded. All positions inside the structure are recordedand calculated as relative to the recognizable elements. Inside astructure, such as a house, a vehicle, a container, a warehouseadditional data or information that may include a written description,an image, a sound recording, or a video recording is associated with aspecific location or a specific location volume determined by locationcoordinates which may be relative position coordinates. The data may beretrieved from the database based on a preset location volume and stepsas provided herein.

One way to visualize this is to assume a house on a hill, including abedroom, a kitchen and a bathroom and with a wall and a garden behindthe house. Images are recorded of the rooms and garden with positions orlocation volumes. A user with a smartphone or tablet in accordance withan embodiment of the present invention sets a location volume thatcaptures a particular room or garden and points a camera in thesmartphone at the front door of the house on the hill. Positionalcoordinates of the front door are determined and a location volume forsearch is calculated. Based on the location volume images, and/or soundsand/or descriptions or documents of the room or garden captured by thelocation volume are retrieved from the database and are displayed on thescreen of the smartphone. A preset location volume is associated with alabel like kitchen and garden and a search location volume is calculatedbased on preset volume, pose and coordinates of the camera. A locationvolume does not need to include the point at which the camera isdirected as shown as 5709 in FIG. 57.

Aspects of the present invention have been described above in connectionwith the embodiments thereof. The embodiments are illustrative examples.It is recognized by those skilled in the art that various modificationsare possible with regard to the combinations of the components and theprocesses of the embodiments and such modifications fall within thescope of the various aspects of the present invention.

The invention claimed is:
 1. Apparatus to determine position coordinatesincluding an altitude of an object, comprising: a mobile, portable andwireless computing device including a camera and positional sensors todetermine positional data of the camera with a lens directed at theobject, the positional data including an altitude and a directionalpose, and at least one range-finding device to determine a distance fromthe mobile, portable and wireless computing device to the object, therange-finding device being selected from the group consisting of laserbased range finder, image based range finder, radar based range-finderand sound based range finder, the object obscuring an obscured objectfrom view by the camera; a processor in the mobile; portable andwireless computing device enabled to determine coordinates including analtitude of the object, based on the positional data including thedirectional pose of the camera and on a distance from the mobile,portable and wireless computing device to the object; a database thatstores data of a plurality of objects and the database is searchable bypositional coordinates including an altitude; the processor enabled todo a search in the database based on a pre-set location volume oflimited size that is based on the coordinates including the altitude ofthe object and the pre-set location volume including a space that isobscured from the camera by the object; and the processor enabled todisplay on a screen data of the obscured object that is identified inthe database based on the search as being located at least partially inthe pre-set location volume.
 2. The apparatus of claim 1, wherein themobile, portable and wireless computing device is a smartphone.
 3. Theapparatus of claim 1, wherein the mobile, portable and wirelesscomputing device is a tablet computer.
 4. The apparatus of claim 1,wherein the camera is attached to an edge of a body of the mobile,portable and wireless computing device.
 5. The apparatus of claim 1,wherein the mobile, portable and wireless computing device has a bodyand the camera is located in a separate body that is enabled to berotatably moved relative to the body of the mobile, portable andwireless computing device.
 6. The apparatus of claim 1, furthercomprising: at least one of the one or more objects located at leastpartially in the location volume having a positioning device with anidentification code.
 7. The apparatus of claim 1, wherein: a size of thelocation volume is based on a size of the object.
 8. The apparatus ofclaim 1, wherein: a size of the location volume is larger than a size ofthe object.
 9. The apparatus of claim 1, wherein: the object obscuresvisibility of at least one other object.
 10. The apparatus of claim 1,wherein: the object is a walled space.
 11. The apparatus of claim 1,wherein: the object is a shipping container.
 12. The apparatus of claim1, wherein: the object is a vehicle.
 13. The apparatus of claim 1,wherein: the object is a living creature.
 14. A method for determiningpositional coordinates including an altitude of an object, comprising:determining by a processor in a computing device, with data from one ormore positional sensors, positional data including an altitude of acomputing device including a camera and including a directional pose ofthe camera with a lens directed at the object, the object obscuring anobscured object from view by the camera; determining by the processor adistance substantially between the camera and the object; determining bythe processor positional coordinates including an altitude of theobject, using at least the positional data including the altitude of thecomputing device, the directional pose of the camera directed at theobject and the distance substantially between the camera and the object;determining coordinates of a location volume of limited size that isbased on the positional coordinates of the object, the coordinates ofthe location volume of limited size including a space that is obscuredfrom the camera b the object; searching a database that stores data of aplurality of objects and the database is searchable by positionalcoordinates including an altitude, for one or more objects that are atleast partially located within the coordinates of the location volume;and displaying on a screen data of an object that is identified in thedatabase as being located at least partially in the location volume as aresult of the searching.
 15. The method of claim 14, wherein thecomputing device is a smartphone or a tablet computer.
 16. The method ofclaim 14, wherein the distance is determined by applying one device ofthe group consisting of a laser based range finder, an image based rangefinder, a radar based range finder and a sound based range finder. 17.The method of claim 14, wherein the camera is attached to an edge of abody of the computing device.
 18. The method of claim 14, wherein thecomputing device has a body and the camera is located in a separate bodythat is enabled to be rotatably moved relative to the body of thecomputing device.
 19. The method of claim 14, wherein: the objectobscures a visibility of at least one other object.
 20. The method ofclaim 14, wherein: the object is a walled space.